Posts

ChatGPT Facts

Image
We have heard a lot about the capabilities of Chat GPT. This post is not about what it can do or how you can use it but some details on what went into creating it GPT-1 was trained on 40 GB of text with 117 million parameter. GPT-2 had 1.5 billion parameters and 40 GB of text was trained on 256 GPUs for several weeks GPT-3 was trained on 175 billion parameters and 45 terabytes of text, making it the largest model ever built. Thats a staggering amount of data. This means ability to respond to a wide range of questions. GPT-3 was trained using both GPUs and TPUs making it computationally very expensive to the tune of millions of dollars just for setting up the infrastructure. Also the continued cost involved in re-training, curating the dataset and so on. This is a significant amount of investment in R&D.  The model was based on unsupervised training which took several months to complete The architecture was built based on a deep neural network called a transformer  first in...

AppsFlyer Cost ETL to S3 and breaking it down further

Image
AppsFlyer, one of the most popular MMP, provides access to APIs to pull attribution and analytics reports from your code. This is particularly useful when you have internal analytics tools and want to gain deeper insights into some metrics. While the reports provide an aggregated view, AppsFlyer also lets you view a granular breakdown of each transaction, in the form of Cost ETL .  You can setup Cost ETL and push to S3 as shown in the link above and the batches get pushed to your bucket automatically each day after the integration is complete. What is tricky is to partition by date and be able to process it. Hopefully this script in Python allows you to do that or at-least help you get started with it. It reads a parquet and partitions by date. As new reports get pushed, the partition date is overwritten to reflect the new data as recommended by AppsFlyer. import pandas as pd import os,sys, json def partitionRawFilesIntoDateParquets (): # Assuming the fourth batch is availabl...

S3 Functions in Python using Boto

 1.  Copy from one folder in a bucket to another using Boto. Configure Boto AWS as mentioned in the documentation  here import boto3 import os,sys bucketName= "YOUR BUCKET NAME" # set this to True if you want the objects in copyToPathDirectory to be deleted first # set to False if the object being copied has the same name deleteObjectsFirst = False # Assuming Boto is setup s3 = boto3.resource( 's3' ) s3Client = boto3.client( 's3' ) bucket = s3.Bucket(bucketName) copyToPathDirectory = "the/directory/youwant/to/copyto/" copyFromPathNames = [ "path1/to/copyfrom/" , "path2/to/copyfrom/" , "and/so/on/" ] # WARNING: This will delete all objects in your folder # Please check your path before running if deleteObjectsFirst: delete_key_list = [] for copyFromPath in copyFromPathNames: for obj in bucket.objects. filter (Prefix = copyFromPath): deleteKey = copyToPathDirectory + obj.key delete_key_list...

Chatbot application with React and Node

https://medium.com/p/setting-up-a-simple-client-server-chatbot-application-using-react-and-node-45b3c7bd5235

5 javascript oops!

https://medium.com/p/5-javascript-oops-9c4ecdcca781

.Net TripleDES encryption not matching in Javascript

https://medium.com/p/net-tripledes-encryption-not-matching-in-javascript-f8c7037f593a

Adding react native admob in your app

https://medium.com/p/adding-react-native-admob-in-your-app-c32f3be2de2d

Reading weights from a specific layer in Keras Model

Image
Follow my video below to know how to read weights for a specific layer in a Keras model. Link to the code sample is here . 

MySQL in Node.js

Image
You can follow my video below to connect to MySQL from Node.js Code sample here .

Finding the version of a python package

pip list | grep [package name]

Finding the location of all pip installed packages

We all agree that it is convenient to install packages using pip . But where do these packages get installed? Which directory? You can find this in many ways, but two of the quickest include using the inspect module or pip show . inspect module: import inspect from class import classname inspect(classname) The third line will give you the full path pip show: pip show [package name] will do the trick too.

Swallow - My next game

Image
Guide the fireball (orange block) to the hole (blue block). Simple! Download the project here (5.2.1f1 and above). It uses the UUEX framework mentioned in the previous post.

Unity3D / C# - Tetris 3D

Image
I could not find any reference for Tetris game anywhere :-(. So, I decided to write one and here it is. This is in Unity3D and is 3D. The only thing missing is the J piece.... just lazy!! This version has no gravity. This means you will see blocks hanging after rows are cleared.  I tried making things easy by attaching Rigidbody to the tiles but they create some nasty side effects. You need to freeze rotation, add physics materials for zero friction and what not. I removed them and used a simply 2D grid instead. A total effort of 15 hours. This is not a full blown game. No bonus, no score, no levels.. just plain and simple game play. Uses a PoolManager for optimising the spawning of blocks and the UUEX framework in my previous post for a start and end UI. And yes, a background music and an effect when a row is cleared. Feel free to adapt it anyway you want. Download the Unity3D project here (version 5.2.1f1 and above). 

A framework on top of the Unity 5 UI system - UUEX v1.0

After a long shot at this, I finally feel I am ready to ship the first cut of this sucker. It's a framework built on top of the UGUI system. I spent a lot of time building a framework on top of NGUI, but with new revisions and fixes, we always ended up doing a lot of patch work. End result.. bloated code. And then came UGUI, which was more or less similar to NGUI, but without the crazy gizmos, editor scripts and what not. It just seemed much slimmer, cleaner and nicer than NGUI. So, I decided to write a framework around it. The framework glues all the elements of UGUI nicely. You don't have to be worried about Canvas, Graphic, MaskableGraphic etc. What is available to you is, SetVisibility, SetTexture, SetSprite which is more neater and friendly. Download the zip and explore it. Let me know what you think. Download UUEX 1.0

Web Crawling using Apache Nutch + MySQL + Solr

If you have come here, then you probably know what Nutch is and you are looking for a test to integrate it with MySQL and Solr or Elasticsearch. So, lets get started. I am not going to guide you with the steps, but give you this awesome link which helped me run a test crawl in couple of hours. Technically, it should have taken not more than half hour, but I had to spend more time trying to figure out the issues mentioned below on my Mac - El Capitan. Firstly follow the steps mentioned in this link.  https://anil.io/post/92/apache-nutch-2-2-mysql-and-solr-5-2-1-tutorial In the process of trying to make things work, you may come across the following issues. I have provided solution for each below 1) Don't ignore this step [STEP 2 in the link above] export JAVA_HOME = "$(/usr/libexec/java_home -v 1.8)" export NUTCH_JAVA_HOME = "$(/usr/libexec/java_home -v 1.8)" 2) Issue when creating the database nutch in MySQL. [STEP 3. 2 in the li...