Friday, July 3, 2020

How to access the local files externally through ngrok and python HTTP Server?

This tutorial explains how to share or access the local files externally through ngrok and Python http server.
I have the below two scenarios
  • a set of files that needs to be shared outside
  • share a simple website externally
The above scenarios can be achieved through Apache Server and external DNS configurations but require more configurations efforts.
  • ngrok free or paid version
  • Python latest version
Let us see how to enable the above scenarios through ngrok and python HTTP server with minimal configuration effort.
As a first step download ngrok(ngrok.com) for the required OS and extract the files
ngrok
ngrok
The ngrok allows you to expose a webserver running on your local machine to the internet. Just tell ngrok what port your web server is listening on.
On the free plan, ngrok’s URLs are randomly generated and temporary. If you want to use the same URL every time, you need to upgrade to a paid plan so that you can use the subdomain option for a stable URL. There are different paid plans they will provide some advance features like custom/reserved domains and multiple tunnels etc
ngrok

How ngrok works

You download and run a program on your machine and provide it the port of a network service, usually a web server.
It connects to the ngrok cloud service which accepts traffic on a public address and relays that traffic through to the ngrok process running on your machine and then on to the local address you specified.
ngrok

Python HTTP Server


Python standard library comes with an in-built webserver which can be invoked for simple web client server communication
The required port number can be assigned and the web server is accessed through this port
My system has python version 3.7.0 installed— “py -vi”
In the first scenario, i want to share some regular files externally
share files externally through ngrok and python http server
To start the HTTP server, cd to to the folder that should be shared(C:\Albin\blogData\demo\Share) through command prompt and execute the command “py -m http.server 80” — change the port number as required
share files externally through ngrok and python http server
Now the files are accessible through localhost
share files externally through ngrok and python http server

Let's now start ngrok to share this folder externally, cd to the folder where ngrok was extracted(C:\Albin\SW\ngrok-stable-windows-amd64)
Execute “ngrok.exe http 80”(80 is where python HTTP server running)
share files externally through ngrok and python http server
Now the external requests(http/https) are forwarded to localhost webserver through ngrok proxy domain
share files externally through ngrok and python http server

The local folders can be shared directly without a HTTP server through inbuilt ngrok file server. To share the local folder directly through ngrok , as a first step configure the authtoken to the ngrok

The authtoken can be retrieved through ngrok dashboard — the user should signup for a account , copy the command to set the authtoken by navigating to the dashboard.

Execute the command

ngrok-auth-token

Start the ngrok process — e.g ngrok http “file:///C:\Albin\blogData\demo\blogproject.blogproject

ngrok-auth-token

Now the files under the specific folder is accessible externally

ngrok-auth-token
Let us now see how to enable the second scenario, to access simple website externally, created a index.html file along with some test files into a folder(C:\Albin\blogData\demo\site), cd to the folder where the index.html and other files are located
share files externally through ngrok and python http server
Re-start the HTTP server, the pages are now accessible outside
share files externally through ngrok and python http server
The traffic can be monitored through the following URL — http://127.0.0.1:4040
share files externally through ngrok and python http server
The ngrok and python HTTP server can be used to share the local files externally without much effort and configurations.



Thursday, July 2, 2020

Geo Location Based Redirects with CloudFront and Apache | Redirect Web traffic Based on Country of Origin with CloudFront

This tutorial explain the approach to enable Geo Location based redirects with CloudFront and Apache.

Geo IP based redirection

Geo IP based redirection is the process of automatically redirecting a website visitor by their geolocation.

There are multiple options to enable the location based redirects in Apache, one of the option is using Geo IP database like MaxMind Geo IP database to map users’s IP to their location. Maxmind Geo IP database can be enabled through Apache module.

If you are using any of the CDN e.g CloudFront provides specific headers with request location, CloudFront will detect the user’s country of origin and pass along the county code to origin server in the CloudFront-Viewer-Country header. You can use this information to customize your responses e.g redirecting the users to specific URL based on origin country.

Prerequisites

Website enabled with CloudFront CDN and Apache

CloudFront Configurations

As a first step white list the CloudFront-Viewer-Country header in Cloudfront distribution

Access <<CloudFront Distribution>> →Behaviors → <<Specific Behavior>>

Edit the behavior

Whitelist CloudFront-Viewer-Country header — the header with user’s country of origin will be sent to origin server(Apache) on every request.

CloudFront-Viewer-Country header will have the two letter country code based on the request origin.

Enable Apache Redirect

Let us now enable the required redirect configuration to virtualhost, add the below redirect rules to enable the the country specific redirects

<VirtualHost *:80>

ServerAdmin [email protected]
DocumentRoot "C:\opt\communique\dispatcher\cache"
ServerName test.albinsblog.com
ServerAlias localhost

RewriteEngine On

RewriteCond %{REQUEST_URI} ^/content/we-retail.html
RewriteCond %{HTTP:CLOUDFRONT-VIEWER-COUNTRY} ^US$
RewriteRule ^.*$ https://test.albinsblog.com/content/we-retail/us/en.html [R=302,L]
RewriteCond %{REQUEST_URI} ^/content/we-retail.html
RewriteCond %{HTTP:CLOUDFRONT-VIEWER-COUNTRY} ^IT$
RewriteRule ^.*$ https://test.albinsblog.com/content/we-retail/it/it.html [R=302,L]
RewriteCond %{REQUEST_URI} ^/content/we-retail.html
RewriteCond %{HTTP:CLOUDFRONT-VIEWER-COUNTRY} ^CA
RewriteRule ^.*$ https://test.albinsblog.com/content/we-retail/ca/en.html [R=302,L]
RewriteCond %{REQUEST_URI} ^/content/we-retail.html
RewriteCond %{HTTP:CLOUDFRONT-VIEWER-COUNTRY} ^FR$
RewriteRule ^.*$ https://test.albinsblog.com/content/we-retail/fr/fr.html [R=302,L]
<Directory />
Options Indexes FollowSymLinks Includes
# Set includes to process .html files
AddOutputFilter INCLUDES .html
AddOutputFilterByType INCLUDES text/html
AllowOverride None
</Directory>

</VirtualHost>

I am using some VPN tool to initiate the connection from different origin country.

Connected the VPN to Canada

Now the user is redirected to Canada specific URL

The user is redirected to the country specific URL based on the users country of origin, CloudFront will detect the user’s country of origin and pass along the county code to origin server(Apache) in the CloudFront-Viewer-Country header. The Apache server redirect the user to the country specific URL’s based on the country code values in CloudFront-Viewer-Country header.





Monday, June 29, 2020

How to configure Dynamic Facets in Adobe Search and Promote?

Dynamic Facets

The dynamic facet is used to create new range selections automatically at the time of search. The facets are included dynamically based on the search result.

The Dynamic Facets feature is not enabled in Adobe Search&Promote, by default. Contact Technical Support to activate the feature for your use.

In our previous tutorial, we have seen how to enable static facet for the search results. Refer the below link for the details, the steps are going to be same for Dynamic facet with small changes.

Facets that are sparsely populated across your website and only appear for a subset of searches are good candidates to make dynamic.

In our example, the product with type “Watch” will have an additional attribute with name “size” and the facet associated with “size” is applicable only for the searches with keyword “Watch”.

The search with key word “Watch” will shows two facets “productType”(static) and “size”(dynamic) but the search with “Book” will shows only “productType”(static) facet.

adobe-search-and-promote-dynamic-facet

Configuring Dynamic Facet

Some additional configuration required to enable Dynamic Facet compared to Static Facet(Refer Static facet tutorial for basic configurations)

Enabled additional product attribute to the feed file based on the product Type — “size”, “size” attribute is applicable only for the productType “Watch”

<feed xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0">
<channel>
<title>Product Feed</title>
<Item>
<link>https://www.example.com/product-title/p/prod1</link>
<title>
<![CDATA[Book Prod1 Title]]>
</title>
<description>
<![CDATA[<p>Prod1 description</p>]]>
</description>
<productType>Book</productType>
<ProductId>prod1</ProductId>
<imageUrl>/content/dam/Images/product/prod1.jpg</imageUrl>
</Item>
<Item>
<link>https://www.example.com/product-title/p/prod2</link>
<title>
<![CDATA[Book Prod2 title]]>
</title>
<description>
<![CDATA[<p>Prod2 description</p>]]>
</description>
<productType>Book</productType>
<ProductId>prod2</ProductId>
<imageUrl>/content/dam/Images/product/prod2.jpg</imageUrl>
</Item>
<Item>
<link>https://www.example.com/product-title/p/prod3</link>
<title>
<![CDATA[Watch Prod3 Title]]>
</title>
<description>
<![CDATA[<p>Prod3 description</p>]]>
</description>
<productType>Watch</productType>
<size>10</size>

<ProductId>prod3</ProductId>
<imageUrl>/content/dam/Images/product/prod3.jpg</imageUrl>
</Item>
<Item>
<link>https://www.example.com/product-title/p/prod4</link>
<title>
<![CDATA[Watch Prod4 Title]]>
</title>
<description>
<![CDATA[<p>Prod4 description</p>]]>
</description>
<productType>Watch</productType>
<size>20</size>

<ProductId>prod4</ProductId>
<imageUrl>/content/dam/Images/product/prod4.jpg</imageUrl>
</Item>
</channel>
</feed>

Create a new meta data definition for “size” field, Settings → Metadata →Definitions

Enable Dynamic Facet option for the metadata

Update the IndexConnector configurations with new meta data field “size”, Settings → Crawling →Index Connector

adobe-search-and-promote-dynamic-facet

Configure facet with name “size” — there is no “Dynamic Facet” setting in Facet, only the configuration is in the underlying metadata configuration(already enabled), Design →Navigation →Facets

adobe-search-and-promote-dynamic-facet

Configure a new Query Cleaning rule, Rules → Query Cleaning , to set the backend parameter “sp_sfvl_df_count”, the sp_sfvl_df_count parameter
determines the total number of dynamic facet fields to return.

adobe-search-and-promote-dynamic-facet

Update the back end transport with Dynamic Facet Support


<search-content-type-header charset="UTF-8">
{ 
	"general": {
        "query" : "<search-query />",		
        "total" : "<search-total />",
        "lower" : "<search-lower />",
        "upper" : "<search-upper />"

	},
	"facets" : [
        {
            "name" : "productType",
            "values" : [<search-field-value-list name="productType" quotes="yes"  data="values" sortby="values" encoding="json" />],
            "counts" : [<search-field-value-list name="productType" quotes="no"  data="results" sortby="values" />]
        }<search-dynamic-facet-fields>,
        {
            "name" : "<search-dynamic-facet-field-name>",
            "dynamic-facet" : 1,
            "values" : 
                    [<search-field-value-list quotes="yes" commas="yes" data="values" sortby="values" encoding="json" />],
                    "counts" : [<search-field-value-list quotes="yes" commas="yes" data="results" sortby="values" />]
        }</search-dynamic-facet-fields>
    ],

	"results" : [
		<search-results>
		{
			"fields" :
			[
				{
					"name" : "mdi",
					"value" : "<search-display-field name="mdi" length="500"  encoding="json" />"
				},

				{
					"name" : "title",
					"value" : "<search-display-field name="title" encoding="json" />"
				},
				{
					"name" : "productType",
					"value" : "<search-display-field name="productType" encoding="json" />"
				},
				{
					"name" : "size",
					"value" : "<search-display-field name="size" encoding="json" />"
				}
            ]		
        } 
        <search-if-not-last>,</search-if-not-last>
		</search-results>
	]
}

The configurations are ready, let us now run a Stage indexing (note the configuration are not pushed to live yet)

Index →Full Index →Staged Index →Run Full Index

adobe-search-and-promote-dynamic-facet

Access http://stage-xxxxxxxxxxx.guided.ss-omtrdc.net/do=json&sp_staged=1&sp_q=Watch

xxxxxxxxxxx — Search and Promote account number

This will respond with JSON data which contains the required Static, Dynamic facet and meta data details.

{
"general": {
"query": "",
"total": "2",
"page_lower": "1",
"page_upper": "2",
"page_total": "1"
},
"facets": [
{
"label": "productType",
"long": false,
"values": [
{
"value": "Watch",
"selected": "false",
"count": "2",
"link": "?do=json;i=1;q1=Watch;sp_q=Watch;sp_staged=1;x1=productType",
"undolink": "",
"threshold": false
}
]
},
{
"label": "size",
"long": false,
"values": [
{
"value": "10",
"selected": "false",
"count": "1",
"link": "?do=json;i=1;q1=10;sp_q=Watch;sp_staged=1;x1=size",
"undolink": "",
"threshold": false
},
{
"value": "20",
"selected": "false",
"count": "1",
"link": "?do=json;i=1;q1=20;sp_q=Watch;sp_staged=1;x1=size",
"undolink": "",
"threshold": false
}
]
}
]
,
"results": [
{
"index": "",
"title": "Watch Prod3 Title",
"productType": "Watch",
"size": "10"
},
{
"index": "",
"title": "Watch Prod4 Title",
"productType": "Watch",
"size": "20"
}
]
}

The configuration can be pushed live after successful validation and run a live index →Full Index →Live Index →Run Full Index

adobe-search-and-promote-dynamic-facet

The URL to access live data http://xxxxxxxxxxx.guided.ss-omtrdc.net/do=json&sp_q=Watch

The facet data in the response can be used to present the filtering options to users to narrow down the website search. The Dynamic Facet option enables the facets based on the search data.



Sunday, June 28, 2020

How to implement autocompletion and search suggestion in AEM through Lucene | Predictive Search in AEM | AEM Search Suggestions

How to implement autocompletion and search suggestion in AEM through Lucene | Predictive Search in AEM | AEM Search Suggestions


This tutorial explain the approach to implement autocompletion and search suggestion in AEM through Lucene.

When you start typing something in search form most of the applications helps you by suggesting the data matching to your search term.

aem-autocompletion-search-suggestion

The purpose of autocomplete is to resolve a partial query , i.e., to search within a controlled vocabulary for items matching a given character string.

Starting from AEM 6.1 the feature of suggestion is available through the suggest module of Lucene. Prior to AEM 6.1, all the possible combination of the words needs to be indexed to support the autocompletion.

The Lucene Suggest module provides a dedicated and optimized data structure allows the engine to give autocompletion and suggestion feature without indexing all the possible combination of a word. 

There is a specific analyzer (AnalyzingInfixSuggester) used that loads the completion values from the indexed data and then build the optimized structure in memory for a fast lookup. 

In order to implements the autosuggestion, feature you need to define an index of type Lucene and for each property X of nodes that you are indexing you can add a specific property useInSuggest to tell to the engine to use X for suggesting query to the user.

Refer the following URL for details on enabling custom index - https://www.albinsblog.com/2020/04/oak-lucene-index-improve-query-in-aem-configure-lucene-index.html  

I have already enabled a custom Lucene index(testindex) for the property "id", add a property "useInSuggest" to tell the engine to use id for suggesting query to the user.

aem-autocompletion-search-suggestion


An additional property suggestUpdateFrequencyMinutes define the frequency of updating the indexed suggestions - useful to mitigate performance issues that can arise if indexed properties are frequently updated by the users of your application. The default value is 10 minutes but the values can be modified as required.

To enable the property "suggestUpdateFrequencyMinutes ", create a node with name "suggest" of type "nt:unstructured" under "testindex" and update the value as required

aem-autocompletion-search-suggestion


 In order to use Lucene index to perform search suggestions, the index definition node (the one of type oak:QueryIndexDefinition) needs to have the compatVersion set to 2. 

aem-autocompletion-search-suggestion


Let us now execute the query to find the suggestions - either one of the below query can be used.

The testindex was defined for the content path "/content/sampledata" so the query will be executed based on the "testindex" but the index name is explicitly defined in the first query.

aem-autocompletion-search-suggestion


SELECT [rep:suggest()] FROM [nt:unstructured] WHERE SUGGEST('te') OPTION(INDEX NAME [testindex]) /* oak-internal */ 

SELECT [rep:suggest()] FROM [nt:unstructured] WHERE SUGGEST('te') AND ISDESCENDANTNODE('/content/sampledata')

The above query uses path restriction to filter the data, it requires evaluatePathRestrictions property should enabled as true on index definition.

aem-autocompletion-search-suggestion


The Query tool shows the total number of unique suggestions matching with the search data but it wont displays the matching node details

aem-autocompletion-search-suggestion


The below Servlet can be used to fetch the suggestion data through QueryManager API
import java.io.IOException;

import javax.jcr.RepositoryException;
import javax.jcr.Session;
import javax.jcr.query.InvalidQueryException;
import javax.jcr.query.Query;
import javax.jcr.query.QueryManager;
import javax.jcr.query.QueryResult;
import javax.jcr.query.Row;
import javax.jcr.query.RowIterator;
import javax.servlet.Servlet;
import javax.servlet.ServletException;

import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.servlets.HttpConstants;
import org.apache.sling.api.servlets.SlingSafeMethodsServlet;
import org.json.JSONArray;
import org.osgi.framework.Constants;
import org.osgi.service.component.annotations.Component;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Component(immediate = true, service = Servlet.class, property = {
		Constants.SERVICE_DESCRIPTION + "=Custom Root Mapping", "sling.servlet.methods=" + HttpConstants.METHOD_GET,
		"sling.servlet.paths=" + "/bin/getSuggestions", "service.ranking=" + 100001 })
public class SuggestionData extends SlingSafeMethodsServlet {

	/**
	* 
	*/
	private static final long serialVersionUID = 1L;

	protected void doGet(final SlingHttpServletRequest req, final SlingHttpServletResponse resp)
			throws ServletException, IOException {

		Logger logger = LoggerFactory.getLogger(this.getClass());
		logger.error("inside custom servlet");

		final Session session = req.getResourceResolver().adaptTo(Session.class);

		final JSONArray suggestions = new JSONArray();

		String queryString = "SELECT [rep:suggest()]  FROM [nt:unstructured] WHERE "
							 +"SUGGEST('te') OPTION(INDEX NAME [testindex]) /* oak-internal */ ";

		try {
			QueryManager queryManager = session.getWorkspace().getQueryManager();
			Query query = queryManager.createQuery(queryString, Query.JCR_SQL2);
			QueryResult result = query.execute();
			RowIterator rows = result.getRows();

			while (rows.hasNext()) {
				suggestions.put(((Row) rows.next()).getValue("rep:suggest()").getString());
			}

		} catch (InvalidQueryException e) { // TODO Auto-generated catch block
			e.printStackTrace();
		} catch (RepositoryException e) { // TODO Auto-generated
			e.printStackTrace();
		} finally {
			session.logout();

		}

		resp.setContentType("application/json");
		resp.getWriter().write(suggestions.toString());

	}

}

aem-autocompletion-search-suggestion


This suggestion data cab be used to display  the search autocompletion/suggest data to the website users.