Setting up an Overpass API server with Docker

Machine Learning Applied to Stock Market Prediction: A Comparison between LSTM and ESN

5 minute read

Published: October 03, 2022

I have been working with various neural networks for a while and always find recurrent neural network (RNN) very special. Whenever, there is a dependency with previous data points, then these networks shines out. Time-series problem especially fits here for e.g. predicting the weather pattern or stock market. In the past, I have done work to compare various RNNs for such tasks and my work concluded that ESN works quite well compared to simple deep neural network and some RNNs like LSTM or GRU (paper-1, paper-2 and paper-3) (of course this statement vary with the domain). Those works were done mostly with the scientific data for turbulent flow for thermal plumes, which has significant effects on weather (Wikipedia). Now, I was curious to see how these network can work for practical purpose.

Deep learning on HPC cluster with LSF queue

2 minute read

Published: September 28, 2022

This is rather a short documentation to run the TensorFlow jobs on a high performance computing (HPC) cluster which is using LSF Load Sharing Facility. This can be extrapolated to other HPC systems with some tweaks. If you’re new to LSF and HPC jobs with DL then this post nicely summerizes the jargon.

Getting started with MLFlow with MinIO [In 2022]

5 minute read

Published: September 09, 2022

First impression with Firebase Realtime Database using Python [and also with Swift for iOS]

6 minute read

Published: May 14, 2022

[Updated on 05.08.2022]

Firebase is a Backend-as-a-Service (BaaS) app development platform backed by Google. It provides a variety of services like database, cloud storage etc. Most prominent use case (atleast for me) is having a realtime database in almost no time to enable quick prototyping. Recently, I’ve experimented with Python and Swift (iOS) SDKs for Firebase and this documentation summerizes the same.

Firebase in Python

Adding streaming data to Firebase via Python

The intention behind adding the continous stream of data to Firebase is inspired from Internet of Thing (IoT) sensor where data is keep coming, and based upon the data we might want to trigger some action. There are few post already available like this, this and this. As the most simplest case, my target is to aquire some timeseries data on a given interval and send this to Firebase continously. As no external sensor is attached to my system, so I decide to get my sysetm information like RAM usage, CPU usage etc. on a given frequency and then add this to our database.

Creating Firebase project

Creating a Firebase account and setting up database is straightforwad with google id. Here, we will use Firebase Realtime Database.

1. Go to Firebase Console and click on Create a Project. Remember each project is a separate container and it can have several databases as per our requirement, so kind of namespace.
2. Set a desired name of the project and move forward.
3. Depending on the tracking and analysing use case, we can keep the Google Analytics on, and proceed. Depending upon next option will be Google Analytics ID.
4. Now, our project is comissioned and we have our dashboard.
5. On the left sidebar, click on Build>Realtime Database. Then click on Create a Database and select geographical location accordingly.
6. Select on default locked mode which we can change afterwards easily.
7. Now, our realtime database is ready. And we can see the Data, Rules, Backups and Usage tab. Within the Data tab, we can hover on URL and by clicking +, we can add the data. However, we will do this via Python.
8. We need to change the access Rules by clicking on that tab and set them to true. It will allow anyone to read/write data! For more details, see the official docs
9. Then we need to register our application. For the same, go to project overview dashboard and from various icons select the one for the Web.
10. Register the app by defining its name. It will generate a JS code and we need to copy the firebaseConfig part dictionary (i.e. content inside the {})and save it as a txt file in our python projetc folder locally as credential.txt.

Accessing via Python

1. After activating virtual or conda environment install pip install pyrebase4. For me, normal pyrebase was throwing an error so this was the easiest solution, I found at that time.
2. Copy the following script and paste into a new python file. Afterwards try it out. Remember to have credential.txt in the same folder.

import pyrebase
import psutil
import platform
import time
import shutil
import datetime

def get_device_name():
    dn = platform.node()
    #dn = ("").join([i.replace('-','') for i in dn])
    return dn

def get_timestamp():
    ts = time.time()
    return datetime.datetime.fromtimestamp(ts).strftime('%d%m%YT%H%M%S')
    
def sys_info():
    disc_usage = shutil.disk_usage("/")                  
    info = {}
    # info["ts"] =  get_timestamp()
    info["ram_usage"] = psutil.virtual_memory()[2]
    info["cpu_usage"] = psutil.cpu_percent()
    info["disk_usage"] = disc_usage[1]/disc_usage[0]*100
    return info 

def read_cred(filename:str)->dict:
    d = {}
    with open(filename) as f:
        for line in f:
            (key, val) = line.split(': "')
            d[key] = val.split('"')[0]
    return d

def initialize_firebase():
    config = read_cred(filename = "credential.txt")
    firebase = pyrebase.initialize_app(config)
    #_auth = firebase.auth()
    return firebase

def add_data_to_firebase(db):    
    data = sys_info()
    #resp = db.push(data)
    resp = db.child(get_device_name()).child(get_timestamp()).set(data)
    return resp


def print_all_data_in_db():
    users = db.child().get()
    print(users.val())

# %%

if __name__ == "__main__":
    db = initialize_firebase().database()
    ctr = 0
    res = []
    while ctr < 3:
        print("Instance: ", ctr)
        res.append(add_data_to_firebase(db))
        time.sleep(2)
        ctr+=1
    print("Done!")

1. This script will add data in neted form similar to a sensor reading. Here, our key becomes the timstamp and each timestamp contains 3 reading. The parent key is the name of our device.
2. A more sophisticated example can be found on my Github.

Firebase in Swift

This part will make a similar attempt with Firebase RT database using the Swift programming language while targetting the iOS development. Steps are similar and intutive as above, but for the sake of completeness:

1. Create a project on Firebase and make it for iOS.
2. From the quick configuration steps fill up the required fill and make sure to enter correct Bundle ID for your iOS app.
3. Download the config file (GoogleService-Info.plist) then drag and drop in Xcode’s file navigator as shown in the official SDK documentation.
4. Correspondingly, add the entries to podfile for Firebase. My podfile looks as following. If you don’t have podfile then download cocoapod and initialize with pod init. With pod install, dependencies will install. Remember, its a big file therefore, it will take some time. Also, I needed to close my Xcode otherwise keep getting some errors.

# Uncomment the next line to define a global platform for your project
 platform :ios, '12.0'

target 'ObjectDetection' do
  # Comment the next line if you're not using Swift and don't want to use dynamic frameworks
  use_frameworks!

  # Pods for ObjectDetection
  pod 'TensorFlowLiteSwift'

  # Pods for firebase
  pod 'FirebaseAuth'
  pod 'FirebaseFirestore'
  pod 'FirebaseDatabase'
end

1. As a next step, we need to initiate Firebase at the beginning of application with all the configuration. Therefore, in AppDelegate.swift, do the following (again suggested in setup process in Firebase):

import UIKit
import FirebaseCore


@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {

  var window: UIWindow?

  func application(_ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions:
      [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
    FirebaseApp.configure()

    return true
  }
}

1. Now, we’re all set to use Firebase SDK. Depending on the usage, we can write/read data to Firebase’s realtime database. In my one of the module, implementation goes as follow:

import FirebaseDatabase
    func writeToFirebase(outputClass:String, outputClassScore: Float){
        let dateString = "SampleDateAndTime"
        let locationData = "SampleLocation"
        let ref = Database.database().reference().child("deviceID/\(deviceID)").child("\(dateString)")
        ref.updateChildValues(["location":locationData,
                               "outputClass":outputClass,
                               "outputClassScore": outputClassScore])
    }

// Get the deviceId
let deviceID = UIDevice.current.identifierForVendor!.uuidString
// Then we call this function with the arguments
writeToFirebase(outputClass:"myOutputClass", outputClassScore:1.0)

This will then write the desired results to the Firebase. For me the use-case was to deploy a ML model on iPhone and then set the relavant data to the Firebase, so that I can query and build relavant dashboard.

Next steps

Explore and examine the following and update the documentation:

• Adding authentication layer in the iOS application.
• Firestore integration for app data e.g. images.

Reach out to me on Instagram for a faster reply!