Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Sun Jul 3 04:22:27

	Need Help_Data Integrations & ETL Best Practices, Tips & Tricks etc by /u/AugustReborn 1 hour ago.
	Data modeling for clickstream/event data by /u/wtfzambo 11 hours ago.
	How long did you work as a jr. Data Engineer before becoming a Data Engineer (in terms of pay/title)? by /u/Own_Neighborhood_231 14 hours ago.
	Am I a data engineer now? by /u/protienbudspromax 16 hours ago.
	Data Engineering Career Path: Step by Step Complete Roadmap by /u/MlTut 16 hours ago.
	Build a Robust Workflow to Visualize Trending GitHub Repositories in Python by Khuyen Tran yesterday
	Kubernetes Ingress Explained by Bernard Brenyah 3 days ago.
	How to Group-By Pandas DataFrames to Compute the Mean by Giorgos Myrianthous 4 days ago.
	Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform - Episode 302 by tmacey 6 days ago.
	Strategies And Tactics For A Successful Master Data Management Implementation - Episode 301 by tmacey 6 days ago.
	The Skeleton of a Data Science Project by Fernando Tadao Ito 8 days ago.
	I Don’t Care How Big Your Data Is. by Barr Moses 8 days ago.
	5 Advanced SQL Concepts You Should Know In 2022 by Suraj Gurav 8 days ago.
	Building a Data Pipeline for My Strava Data by Jack Leitch 11 days ago.
	SQL and Databases Learning: A Guide for Data Scientists and ML Engineers by Ivo Bernardo 11 days ago.
	Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas - Episode 300 by tmacey 13 days ago.
	Level Up Your Data Platform With Active Metadata - Episode 299 by tmacey 13 days ago.
	Enabling Offline Inferences at Uber Scale by Neeraj Dhake 17 days ago.
	Hire And Scale Your Data Team With Intention - Episode 298 by tmacey 20 days ago.
	Discover And De-Clutter Your Unstructured Data With Aparavi - Episode 297 by tmacey 20 days ago.
	Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault - Episode 296 by tmacey 27 days ago.
	Bringing The Modern Data Stack To Everyone With Y42 - Episode 295 by tmacey 27 days ago.
	A Multipurpose Database For Transactions And Analytics To Simplify Your Data Architecture With Singlestore - Episode 294 by tmacey 34 days ago.
	Data Cloud Cost Optimization With Bluesky Data - Episode 293 by tmacey 34 days ago.
	IO in PostgreSQL: Past, Present, Future (Andres Freund) by CMU Database Group 60 days ago.
	RonDB: A Key-Value Store with SQL Capabilities and LATS Properties (Mikael Ronström) by CMU Database Group 65 days ago.
	Velox: An Open-source Unified Execution Engine (Deepak Majeti) by CMU Database Group 72 days ago.
	Presto® on Apache Kafka® At Uber Scale by Yang Yang 79 days ago.
	QuestDB: Fast Open Source Time Series Database (Vlad Ilyushchenko) by CMU Database Group 79 days ago.
	Securing Kafka® Infrastructure at Uber by Prateek Agarwal 86 days ago.
	Yellowbrick: An Elastic Data Warehouse on Kubernetes (Mark Cusack) by CMU Database Group 88 days ago.
	Design and Implementation of the RelationalAI Knowledge Graph Management System (Martin Bravenboer) by CMU Database Group 88 days ago.
	Stardog Query Optimiser: Architecture and Cardinality Estimations for Graph Queries (Pavel Klinov) by CMU Database Group 101 days ago.
	Open-source Change Data Capture With Debezium (Gunnar Morling) by CMU Database Group 108 days ago.
	ApertureDB: Designing a Purpose-built System for Visual Data and Data Science (Vishakha Gupta) by CMU Database Group 114 days ago.
	Orca: A Modular Query Optimizer Architecture for VMware Greenplum (Venkatesh Raghavan) by CMU Database Group 128 days ago.
	RavenDB: Practical Considerations for ACID/MVCC Storage Engines (Oren Eini) by CMU Database Group 132 days ago.
	SpiceDB: Flexible Permissions Database for the Internet Era (Jake Moshenko) by CMU Database Group 133 days ago.
	HTAP with Azure Cosmos DB: Hybrid Transaction & Analytical Processing (Hari Sudan S) by CMU Database Group 135 days ago.
	Project RADAR: Intelligent Early Fraud Detection System with Humans in the Loop by Sergey Zelvenskiy 151 days ago.
	Cost Efficiency @ Scale in Big Data File Format by Xinli Shang 158 days ago.
	Apache Arrow: High-Performance Columnar Data Framework (Wes McKinney) by CMU Database Group 161 days ago.
	How We Build Firebolt (Benjamin Wagner) by CMU Database Group 193 days ago.
	Improving HDFS I/O Utilization for Efficiency by Kevin Cheng 262 days ago.
	Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot by Jacob Tsafatinos 282 days ago.
	YAML Generator for Funnel YAML Files: Streamlining the Mobile Data Workflow Process by Jack Ning 289 days ago.
	Streaming Real-Time Analytics with Redis, AWS Fargate, and Dash Framework by Piyush Choudhary 303 days ago.
	Enabling Seamless Kafka Async Queuing with Consumer Proxy by Yang Yang 305 days ago.
	How Data Shapes the Uber Rider App by Divya Babu Ravichandran 310 days ago.
	Building Scalable Streaming Pipelines for Near Real-Time Features by Feng Xu 312 days ago.
	Efficiently Managing the Supply and Demand on Uber’s Big Data Platform by Zheng Shao 323 days ago.
	Cost-Efficient Open Source Big Data Platform at Uber by Zheng Shao 325 days ago.
	Challenges and Opportunities to Dramatically Reduce the Cost of Uber’s Big Data by Zheng Shao 327 days ago.
	How Uber Achieves Operational Excellence in the Data Quality Experience by Ying Zou 331 days ago.
	Containerizing Apache Hadoop Infrastructure at Uber by Mithun Mathew 345 days ago.
	Analyzing Customer Issues to Improve User Experience by Nimesh Agarwal 351 days ago.
	Automating Merchant Live Monitoring with Real-Time Analytics: Charon by Marco Vita 429 days ago.
	My thoughts on the data mesh by Daniel Abadi 464 days ago.
	Uber’s Journey Toward Better Data Culture From First Principles by Krishna Puttaswamy 473 days ago.
	Turning Metadata Into Insights with Databook by Sunheng Taing 599 days ago.
	Operating Apache Pinot @ Uber Scale by Yupeng Fu 620 days ago.
	Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning by Steffon Davis 760 days ago.
	Monitoring Data Quality at Scale with Statistical Modeling by Ye Henry Li 786 days ago.
	Building a Backtesting Service to Measure Model Performance at Uber-scale by Sam Xiao 870 days ago.
	Women in Data Science at Uber: Moving the World With Data in 2020—and Beyond by Emily Bailey 886 days ago.
	Designing a Production-Ready Kappa Architecture for Timely Data Stream Processing by Amey Chaugule 891 days ago.
	Engineering SQL Support on Apache Pinot at Uber by Haibo Wang 899 days ago.
	Uber Visualization Highlights: Displaying City Street Speed Clusters with SpeedsUp by Bryant Luong 912 days ago.
	It's time to rethink how we share data on the Web by Daniel Abadi 927 days ago.
	Uber’s Data Platform in 2019: Transforming Information to Intelligence by Nikhil Joshi 928 days ago.
	Productionizing Distributed XGBoost to Train Deep Tree Models with Large Data Sets at Uber by Joseph Wang 935 days ago.
	Introducing SLOG: Cheating the low-latency vs. strict serializability tradeoff by Daniel Abadi 940 days ago.
	The dangers of conditional consistency guarantees by Daniel Abadi 940 days ago.
	Introduction to Transaction Isolation Levels by Daniel Abadi 940 days ago.
	Correctness Anomalies Under Serializable Isolation by Daniel Abadi 940 days ago.
	Overview of Consistency Levels in Database Systems by Daniel Abadi 940 days ago.
	An explanation of the difference between Isolation levels vs. Consistency levels by Daniel Abadi 940 days ago.
	Partitioned consensus and its impact on Spanner’s latency by Daniel Abadi 941 days ago.
	It’s Time to Move on from Two Phase Commit by Daniel Abadi 941 days ago.
	NewSQL database systems are failing to guarantee consistency, and I blame Spanner by Daniel Abadi 941 days ago.
	Uber Visualization Highlights: How Urban Symphony Adds an Audio Dimension to Visualization by Jon Sadka 955 days ago.
	Taking City Visualization into the Third Dimension with Point Clouds, 3D Tiles, and deck.gl by Ib Green 977 days ago.
	Building a Better Big Data Architecture: Meet Uber’s Presto Team by Wayne Cunningham 997 days ago.
	Science at Uber: Making a Real-world Impact with Data Science by Molly Vorwerck 1047 days ago.
	Less is More: Engineering Data Warehouse Efficiency with Minimalist Design by Ritesh Agrawal 1053 days ago.
	Science at Uber: Powering Uber’s Ridesharing Technologies Through Mapping by Molly Vorwerck 1061 days ago.
	Science at Uber: Bringing Research to the Roads by Molly Vorwerck 1068 days ago.
	Science at Uber: Building a Data Science Platform at Uber by Wayne Cunningham 1075 days ago.
	Making Apache Spark Effortless for All of Uber by Abhishek Modi 1081 days ago.
	An analysis of the strengths and weaknesses of Apache Arrow by Daniel Abadi 1558 days ago.
	Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? by Daniel Abadi 1705 days ago.
	Hazelcast and the Mythical PA/EC System by Daniel Abadi 1728 days ago.
	Scala Days 2017 - Jepsen Keynote by Aphyr 1886 days ago.
	Distributed consistency at scale: Spanner vs. Calvin by Daniel Abadi 1913 days ago.
	Jepsen: CockroachDB beta-20160829 by Aphyr 1962 days ago.
	Jepsen: MongoDB 3.4.0-rc3 by Aphyr 1971 days ago.
	IEEE Computer issue on the CAP Theorem by Daniel Abadi 2148 days ago.
	Jepsen: VoltDB 6.3 by Aphyr 2181 days ago.
	Jepsen: Crate 0.54.9 version divergence by Aphyr 2195 days ago.
	Jepsen: RethinkDB 2.2.3 reconfiguration by Aphyr 2353 days ago.
	Jepsen: RethinkDB 2.1.5 by Aphyr 2371 days ago.
	Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item by Daniel Abadi 2437 days ago.
	Jepsen: Percona XtraDB Cluster by Aphyr 2493 days ago.
	Jepsen: MariaDB Galera Cluster by Aphyr 2496 days ago.
	Jepsen: Chronos by Aphyr 2518 days ago.
	Jepsen: Aerospike by Aphyr 2616 days ago.
	Jepsen: Elasticsearch 1.5.0 by Aphyr 2623 days ago.
	Jepsen: MongoDB stale reads by Aphyr 2630 days ago.
	Jepsen: Elasticsearch by Aphyr 2939 days ago.
	Jepsen: etcd and Consul by Aphyr 2945 days ago.
	Jepsen: RabbitMQ by Aphyr 2948 days ago.
	Computational techniques in Knossos by Aphyr 2968 days ago.
	Strong consistency models by Aphyr 2970 days ago.
	Jepsen: Redis redux by Aphyr 3126 days ago.
	Jepsen: Strangeloop Hangout by Aphyr 3199 days ago.
	Jepsen: RICON East talk by Aphyr 3199 days ago.
	Jepsen: Cassandra by Aphyr 3203 days ago.
	Jepsen: Kafka by Aphyr 3203 days ago.
	Jepsen: NuoDB by Aphyr 3204 days ago.
	Jepsen: Zookeeper by Aphyr 3204 days ago.
	Automating Jepsen by Aphyr 3291 days ago.
	The network is reliable by Aphyr 3317 days ago.
	Asynchronous replication with failover by Aphyr 3329 days ago.
	Jepsen: final thoughts by Aphyr 3330 days ago.
	Jepsen: Riak by Aphyr 3331 days ago.
	Jepsen: MongoDB by Aphyr 3332 days ago.
	Jepsen: On the perils of network partitions by Aphyr 3332 days ago.
	Jepsen: Redis by Aphyr 3332 days ago.
	Jepsen: Postgres by Aphyr 3332 days ago.
	Is Upstart the right way to get college student start-ups funded? by Daniel Abadi 3524 days ago.
	Defending Matt Welsh’s 'Startup University' Post by Daniel Abadi 3658 days ago.
	If all these new DBMS technologies are so scalable, why are Oracle and DB2 still on top of TPC-C? A roadmap to end their dominance. by Daniel Abadi 3699 days ago.
	Replication and the latency-consistency tradeoff by Daniel Abadi 3859 days ago.
	Overview of the Oracle NoSQL Database by Daniel Abadi 3921 days ago.
	Hadoop's tremendous inefficiency on graph data management (and how to avoid it) by Daniel Abadi 3992 days ago.
	Why Sam Madden is wrong about peer review by Daniel Abadi 4062 days ago.
	Why I'm doing a start-up pre-tenure by Daniel Abadi 4113 days ago.

Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Sun Jul 3 04:22:27