Using a REST API in Python and encapsulate with our own with Flask

3 minute read

Published:

A REpresentational State Transfer (REST) is an application program interface (API) paradigm that employs HTTP requests to GET, PUT, POST, and DELETE data. With request module in Python, it is super easy to use a REST API to get the data from any web service. In addition, with Flask, it’s easy to deploy our own REST API server. This comes handy when we need to fetch some data from the web via available APIs and need to do a few data manipulation before using it finally. Here, I document the usage of both i.e. using an existing API to get the data and then serving the data in the final form with some postprocessing.

For this demo, I am building an API that finds the nearby public toilets from the given coordinates (and radius, if provided) and then gives their distances from the queried point.

Part 1: Getting the data from the web

For getting the basic data, we could directly use OpenStreetMap (OSM) which has a vast amount of information. Overpass query language (OQL) provides a convenient way to query the data from OSM. There are many great tutorials available to get started with OSM and OQL: OQL official and from mapsaregreat. FindToilets class in request_data_osm.py does the work to find the nearby toilets from OSM with OQL. You can get a nice pandas dataframe by just querying as follow:

find_wc = FindToilets(radius=600,lat=12.973132,lon=77.601479).provide_dataframe()

You can read and change as per your taste. The default radius is 100 m with some default coordinates. The fetch_data_from_overpass function is where we request for the data from OSM. As said, after querying the data from OSM, we calculate the distance (straight-line distance) between the queried point and the toilets and return it along with the data of OSM.

Part 2: Setting up our own REST server with Flask

Flask is a web framework in Python and it is fairly easy to set up a server. Before we jump to the code, let’s have a look at the template folder, which contains the basic HTML code and it can be opened in any internet browser to query from our Flask server (once we start running it) like a web-app (very basic app!). main_api.py file has the code for Flask based API. All functions decorated with @app.route are related to Flask and it provides a routing (like homepage and child pages). Our home page simply shows the predefined template which serves as a web-app (alongside our REST service). Our REST api resides at /api/v1/toilets wrt. the homepage.

Once all dependencies have been installed, we can start the rest API from our terminal by running this script. As we are running it in the Debug mode, so all the logs with a lot of other information will be logged in and will be displayed on the terminal. If the server is started successfully, then we can open our home page: http://localhost:5000/, which will be the web-app for us. Here, we can fill out the coordinates and the radius and by pressing the submit button, we will get the data in JSON (geojson to be more precise). As said, we would like to have a programmatic way to access this data via REST, which would be possible with an address like: http://127.0.0.1:5000/api/v1/toilets?src=12.973132,77.601479&radius=500 It will also give us similar data.

You May Also Enjoy

Setting up an Overpass API server with Docker

3 minute read

Published:

You might have already seen my blog on OpenStreetMap from 2020. In that post, I briefly talked about Overpass API server with a pointer to a GitHub repo to setting up locally. However, that setup might fail if we try to build for entire planet. Additionally, source repository hasn’t been updated for 5 years. I’ve received some requests to assist in setting up the Overpass server. Therefore, this short post basically illustrates the process of local Overpass server.

Starting your own Overpass API server

Currently, the process has been very simple and straightforward, thanks to this docker image docker image.

Building the server from raw OSM files

This approach is beneficial when our region of interest in small (compared to entire world).

docker run -e OVERPASS_META=yes -e OVERPASS_MODE=init -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/ -e OVERPASS_RULES_LOAD=10 -v /overpass_db/:/db/overpass_clone_db -p 8888:80 -it --name overpass_monaco wiktorn/overpass-api

This usually takes 5 minutes on a normal computer including the downloading image from dockerhub, downloading OSM file from geofabrik and building the database for a 700 KB file. All the generated builds can be used with the server. At the end of this build, docker container will be stopped and need to be started again. For the same, either one can assign a name to container in the previous step or can look at auto-generated name with docker ps --all command. After grabbing the name, simply start the container as docker start <CONTAINER NAME>. Alternatively, one can pass -e OVERPASS_STOP_AFTER_INIT false option so that we can continue the instance after flushing database.

img

With that, we can query for a pizza shop: -.

In the above, we didn’t use a custom region as shown here obtained from OSM tool without tweaks. But, workaround should be simple, i.e. either providing file as file:/// as mention here or hosting file with local HTTP server.

Note: So far, I am not able to circumvent the issue when I try to mount a local folder in current directory. Although build was successful however during the query server results in error. This is specific to windows only.

Cloning for entire world

When we need to scale up to entire world, then cloning is a better option compared to building from the raw OSM file. In this case, we can pass the option in OVERPASS_MODE for clone. The data will be cloned from the Overpass API server with the defined replication. You can check out available replication frequency on the OSM wiki.

docker run -e OVERPASS_MODE=clone -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/day/ -v /big/docker/overpass_clone_db/:/db -p 8888:80 -it --name overpass_world wiktorn/overpass-api

This process took approx 2 hours with a good internet speed on a Linux machine, and it took approx 204 GB. For sure, this number will grow with more contributions. We can now query for nearby Indian restaurants from our POI with query: -

Using in Python

I have a sample REST API with Flask which basically finds the nearest toilets from the query point. To use the above server is easy, we just need to change the URL of Overpass API server mentioned here to self.sever = 'http://localhost:8888/api/interpreter'.

Reach out to me on Instagram for a faster reply! "Buy Me A Coffee"

Machine Learning Applied to Stock Market Prediction: A Comparison between LSTM and ESN

5 minute read

Published:

I have been working with various neural networks for a while and always find recurrent neural network (RNN) very special. Whenever, there is a dependency with previous data points, then these networks shines out. Time-series problem especially fits here for e.g. predicting the weather pattern or stock market. In the past, I have done work to compare various RNNs for such tasks and my work concluded that ESN works quite well compared to simple deep neural network and some RNNs like LSTM or GRU (paper-1, paper-2 and paper-3) (of course this statement vary with the domain). Those works were done mostly with the scientific data for turbulent flow for thermal plumes, which has significant effects on weather (Wikipedia). Now, I was curious to see how these network can work for practical purpose.

Deep learning on HPC cluster with LSF queue

2 minute read

Published:

This is rather a short documentation to run the TensorFlow jobs on a high performance computing (HPC) cluster which is using LSF Load Sharing Facility. This can be extrapolated to other HPC systems with some tweaks. If you’re new to LSF and HPC jobs with DL then this post nicely summerizes the jargon.