AppsFlyer Cost ETL to S3 and breaking it down further

AppsFlyer, one of the most popular MMP, provides access to APIs to pull attribution and analytics reports from your code. This is particularly useful when you have internal analytics tools and want to gain deeper insights into some metrics. While the reports provide an aggregated view, AppsFlyer also lets you view a granular breakdown of each transaction, in the form of Cost ETL. 



You can setup Cost ETL and push to S3 as shown in the link above and the batches get pushed to your bucket automatically each day after the integration is complete. What is tricky is to partition by date and be able to process it. Hopefully this script in Python allows you to do that or at-least help you get started with it. It reads a parquet and partitions by date. As new reports get pushed, the partition date is overwritten to reflect the new data as recommended by AppsFlyer.

import pandas as pd
import os,sys, json

def partitionRawFilesIntoDateParquets():
# Assuming the fourth batch is available
# And you have downloaded the raw files to your local directory
s3Path = "cost_etl/v1/dt="+fromDate+"/b=4/channel/"
dateFrames = None

arr = os.listdir(s3Path)
for file in arr:
if file.endswith('.parquet'):
df = pd.read_parquet(s3Path+"/"+file)

dates = df["date"].unique()
if dateFrames == None:
dateFrames = dict(list(df.groupby('date')))
else:
currentFrames = dict(list(df.groupby('date')))
for date in currentFrames:
dateFrames[date] = dateFrames[date].append(currentFrames[date])

for date in dateFrames:
content = dateFrames[date]
datePath = "/datePartitions/"+date
if not os.path.exists(datePath):
os.makedirs(datePath)

content.to_parquet("/datePartitions/"+date+"/gz.parquet",compression='gzip')

Comments

Post a Comment

Popular posts from this blog

Authoritative Server - MMO

Latency, Authoritative Server, Lag Compensation, Course Correction, Packet Drops and Client Side Prediction. These are things you would worry about when attempting to build a MMO game. For companies where the delay matters these logics are a must. This post is on setting up an Authoritative Server. There are different ways of setting up the authoritative server. Some of them is mentioned here. You have N clients and one client acts as the authoritative server: Imagine N clients with different machine configurations. The server would need to decide the best possible machine and have the physics simulation run in that. We then have issues with client disconnecting in which case we need to swap the authoritative client to another machine. This means we have to maintain all the state the game was in and move it to the new authoritative client. In an online environment this is simple nothing but nonsense. There is a dedicated server which simulates everything including physics and ...
Read more

Code dependencies

Are you struggling to make a given set of scripts from a project compile error free? If that is true, then you are in the same boat as I am right now. I have come across projects where logic is scattered around to get a functionality out of the pipeline with limited value to object oriented programming, and the most valuable and time saving of all is the plugin architecture. This is as best as it can get. Let me explain.. A producer has an idea and wants to make a quick prototype. He opens a new project, drags and drops the packages he needs and boom the game runs. Obviously you don't expect a producer to code (unless he is from development). The game would not be perfect, but at least it would get the ball rolling and you don't need a developer to assist the producer. Now, to do this, you would want the packages to follow a common rule. You can do this using an interface and have classes implement them. For example, the avatar package and the pet package can implement the...
Read more