Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Tue Dec 1 00:21:35

	Streaming Data Integration Without The Code at Equalum - Episode 161 by tmacey 45 minutes ago.
	Anyone who attempted Cloudera DE575 certification? by /u/redder_ph 3 hours ago.
	Airflow - Saving file to /usr/local/airflow/repo showing /root/ instead, and getting Permission Error by /u/kristiclimbs 4 hours ago.
	Fun Data Engineering Projects by /u/scatterbrained_mugen 8 hours ago.
	What exactly does being proficient in SQL entail? by /u/Severe-Childhood1382 8 hours ago.
	Updating existing SlackClient package in docker container, then getting 'no module named 'slackclient' by /u/kristiclimbs 8 hours ago.
	Hello! can anyone help me with my degree? only takes 2 minutes :) by /u/theresaoctober 10 hours ago.
	Transcribe Your Zoom Meetings For Free with AWS! by /u/twopairisgood 12 hours ago.
	Am I on the right track to building an "open-source"-esque (cheap) Data Lake/ Data Warehouse on AWS? by /u/OneOverNever 12 hours ago.
	Stomach Cancer Death Rate Ranking \| TOP 10 Country from 1990 to 2017 by /u/IndividualReveal8238 13 hours ago.
	Do faang DE interviews even ask about big data tools? Or is it just SQL, python and systems design? by /u/1337codethrow yesterday
	ksqlDB: A Stream-Relational Database System (Matthias J. Sax, Confluent) by CMU Database Group yesterday
	Fauna: Lessons Learned Building a Real World, Calvin-based System (Matt Freels, Fauna) by CMU Database Group yesterday
	Horovod v0.21: Optimizing Network Utilization with Local Gradient Aggregation and Grouped Allreduce by Travis Addair 6 days ago.
	Keeping A Bigeye On The Data Quality Market - Episode 160 by tmacey 7 days ago.
	A Deep Dive into Spark SQL's Catalyst Optimizer (Cheng Lian + Maryann Xue, DataBricks) by CMU Database Group 10 days ago.
	Datometry Hyper-Q: Virtualizing the World's Enterprise Data Warehouses (Lyublena Antova, Datometry) by CMU Database Group 11 days ago.
	Self Service Data Management From Ingest To Insights With Isima - Episode 159 by tmacey 13 days ago.
	EraDB: Designing Systems for Cardinality and Dimensionality (Todd Persen, EraDB) by CMU Database Group 19 days ago.
	Turning Metadata Into Insights with Databook by Sunheng Taing 20 days ago.
	Query Optimization at Snowflake (Jiaqi Yan, SnowflakeDB) by CMU Database Group 20 days ago.
	Building A Cost Effective Data Catalog With Tree Schema - Episode 158 by tmacey 20 days ago.
	Refactoring Query Processing in MySQL (Norvald H. Ryeng, Oracle) by CMU Database Group 24 days ago.
	FoundationDB or: How I Learned to Stop Worrying and Trust the Database (Markus Pilman, Snowflake) by CMU Database Group 24 days ago.
	Add Version Control To Your Data Lake With LakeFS - Episode 157 by tmacey 27 days ago.
	Meet the 2020 Safety Engineering Interns: COVID Edition by Safety Engineering Interns 32 days ago.
	Cloud Native Data Security As Code With Cyral - Episode 156 by tmacey 35 days ago.
	Operating Apache Pinot @ Uber Scale by Yupeng Fu 41 days ago.
	Better Data Quality Through Observability With Monte Carlo - Episode 155 by tmacey 42 days ago.
	Building from the Baltics: Meet the Uber Engineering Team in Vilnius, Lithuania by Ieva Sapokaite 46 days ago.
	Rapid Delivery Of Business Intelligence Using Power BI - Episode 154 by tmacey 49 days ago.
	Apache Arrow Flight: Accelerating Columnar Dataset Transport (Wes McKinney, Ursa Labs) by CMU Database Group 49 days ago.
	Ludwig v0.3 Introduces Hyperparameter Optimization, Transformers and TensorFlow 2 support by Travis Addair 55 days ago.
	Self Service Real Time Data Integration Without The Headaches With Meroxa - Episode 153 by tmacey 56 days ago.
	Revolutionizing Money Movements at Scale with Strong Data Consistency by Aakriti Singla 59 days ago.
	CockroachDB's Query Optimizer (Rebecca Taft, Cockroach Labs) by CMU Database Group 61 days ago.
	Speed Up And Simplify Your Streaming Data Workloads With Red Panda - Episode 152 by tmacey 62 days ago.
	PlanetScale: Query Planning for a Sharded System like Vitess (Sugu Sougoumarane) by CMU Database Group 65 days ago.
	Spearheading Open Source: A Conversation with Jim Jagielski, Staff Technical Program Manager with the Uber Open Source Program Office by Uber Open Source Office 67 days ago.
	ScyllaDB: No-Compromise Performance (Avi Kivity) by CMU Database Group 93 days ago.
	TerminusDB: Building a Native Revision Control DB from Scratch (Gavin Mendel-Gleason) by CMU Database Group 93 days ago.
	YugabyteDB: Bringing Together the Best of Amazon Aurora and Google Spanner (Karthik Ranganathan) by CMU Database Group 97 days ago.
	Splice Machine – An HTAP DB at Scale (Daniel Gómez Ferro & Yi Xia) by CMU Database Group 100 days ago.
	Designing Edge Gateway, Uber’s API Lifecycle Management Platform by Madan Thangavelu 104 days ago.
	Standing for Safety: Meet the Uber Sao Paulo Tech Team by Camila Carvalho 125 days ago.
	Introducing Domain-Oriented Microservice Architecture by Adam Gluck 130 days ago.
	Engineering Failover Handling in Uber’s Mobile Networking Infrastructure by Sivabalan Narayanan 139 days ago.
	Fiber: Distributed Computing for AI Made Simple by Jiale Zhi 153 days ago.
	Editing Massive Geospatial Data Sets with nebula.gl by Georgios Karnas 160 days ago.
	Profiles in Coding: Diana Yanakiev, Uber ATG, Pittsburgh by Bea Schuster 167 days ago.
	Jepsen: On the perils of network partitions by Aphyr 170 days ago.
	Building a Large-scale Transactional Data Lake at Uber Using Apache Hudi by Nishith Agarwal 174 days ago.
	Introducing Neuropod, Uber ATG’s Open Source Deep Learning Inference Engine by Vivek Panyam 175 days ago.
	Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning by Steffon Davis 181 days ago.
	Meta-Graph: Few-Shot Link Prediction Using Meta-Learning by Ankit Jain 185 days ago.
	Profiles in Coding: Christabelle Bosson, Uber Elevate, San Francisco by Bea Schuster 186 days ago.
	Developing the Next Generation of Coders with the Dev/Mission Uber Coding Fellowship by Kurtis Nusbaum 192 days ago.
	Introducing Athenadriver: An Open Source Amazon Athena Database Driver for Go by Henry Fuheng Wu 194 days ago.
	Building Uber’s Go Monorepo with Bazel by Zhongpeng Lin 200 days ago.
	Announcing a New Framework for Designing Optimal Experiments with Pyro by Adam Foster 202 days ago.
	Monitoring Data Quality at Scale with Statistical Modeling by Ye Henry Li 207 days ago.
	Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions by Rui Wang 208 days ago.
	Introducing Piranha: An Open Source Tool to Automatically Delete Stale Code by Murali Krishna Ramanathan 258 days ago.
	Fostering a Culture of Sponsorship: Introducing Uber’s Engineering and Sponsorship Development Program by Molly Vorwerck 263 days ago.
	Why We Leverage Multi-tenancy in Uber’s Microservice Architecture by Amit Gud 264 days ago.
	Celebrating International Women’s Day: Meet the Women Building Uber’s Global Platform by Molly Vorwerck 266 days ago.
	Introducing Uber’s Open Source Principles by Brian Hsieh 270 days ago.
	Under the Hood of Uber ATG’s Machine Learning Infrastructure and Versioning Control Platform for Self-Driving Vehicles by Yu Guo 271 days ago.
	Profiles in Coding: Rick Boone, Core Infrastructure, San Francisco by Wayne Cunningham 277 days ago.
	Tricks of the Trade: Tuning JVM Memory for Large-scale Services by Xinli Shang 278 days ago.
	Building the Future of Mobility from the Pacific Northwest: Meet the Uber Seattle Tech Team by Kara Ortbal 286 days ago.
	Building a Backtesting Service to Measure Model Performance at Uber-scale by Sam Xiao 291 days ago.
	Counting Calories: How We Improved the Performance and Developer Experience of UberEats.com by Chris Lewis 292 days ago.
	Women in Data Science at Uber: Moving the World With Data in 2020—and Beyond by Emily Bailey 307 days ago.
	Designing a Production-Ready Kappa Architecture for Timely Data Stream Processing by Amey Chaugule 312 days ago.
	Engineering SQL Support on Apache Pinot at Uber by Haibo Wang 320 days ago.
	It's time to rethink how we share data on the Web by Daniel Abadi 348 days ago.
	Introducing SLOG: Cheating the low-latency vs. strict serializability tradeoff by Daniel Abadi 361 days ago.
	The dangers of conditional consistency guarantees by Daniel Abadi 361 days ago.
	Introduction to Transaction Isolation Levels by Daniel Abadi 361 days ago.
	Correctness Anomalies Under Serializable Isolation by Daniel Abadi 361 days ago.
	Overview of Consistency Levels in Database Systems by Daniel Abadi 361 days ago.
	An explanation of the difference between Isolation levels vs. Consistency levels by Daniel Abadi 361 days ago.
	Partitioned consensus and its impact on Spanner’s latency by Daniel Abadi 362 days ago.
	It’s Time to Move on from Two Phase Commit by Daniel Abadi 362 days ago.
	NewSQL database systems are failing to guarantee consistency, and I blame Spanner by Daniel Abadi 362 days ago.
	Jepsen: Zookeeper by Aphyr 496 days ago.
	Strong consistency models by Aphyr 829 days ago.
	Jepsen: Aerospike by Aphyr 921 days ago.
	An analysis of the strengths and weaknesses of Apache Arrow by Daniel Abadi 979 days ago.
	Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? by Daniel Abadi 1126 days ago.
	Hazelcast and the Mythical PA/EC System by Daniel Abadi 1149 days ago.
	Jepsen: MongoDB stale reads by Aphyr 1189 days ago.
	Scala Days 2017 - Jepsen Keynote by Aphyr 1307 days ago.
	Distributed consistency at scale: Spanner vs. Calvin by Daniel Abadi 1334 days ago.
	Jepsen: CockroachDB beta-20160829 by Aphyr 1383 days ago.
	Jepsen: MongoDB by Aphyr 1391 days ago.
	Jepsen: MongoDB 3.4.0-rc3 by Aphyr 1392 days ago.
	Jepsen: RethinkDB 2.2.3 reconfiguration by Aphyr 1456 days ago.
	The network is reliable by Aphyr 1477 days ago.
	IEEE Computer issue on the CAP Theorem by Daniel Abadi 1568 days ago.
	Jepsen: Crate 0.54.9 version divergence by Aphyr 1582 days ago.
	Jepsen: VoltDB 6.3 by Aphyr 1601 days ago.
	Jepsen: Cassandra by Aphyr 1623 days ago.
	Asynchronous replication with failover by Aphyr 1664 days ago.
	Jepsen: RethinkDB 2.1.5 by Aphyr 1792 days ago.
	Jepsen: Riak by Aphyr 1821 days ago.
	Jepsen: final thoughts by Aphyr 1821 days ago.
	Jepsen: NuoDB by Aphyr 1821 days ago.
	Jepsen: Kafka by Aphyr 1821 days ago.
	Jepsen: RICON East talk by Aphyr 1821 days ago.
	Jepsen: Strangeloop Hangout by Aphyr 1821 days ago.
	Jepsen: Redis redux by Aphyr 1821 days ago.
	Jepsen: RabbitMQ by Aphyr 1821 days ago.
	Jepsen: etcd and Consul by Aphyr 1821 days ago.
	Jepsen: Elasticsearch by Aphyr 1821 days ago.
	Jepsen: Elasticsearch 1.5.0 by Aphyr 1821 days ago.
	Jepsen: Chronos by Aphyr 1821 days ago.
	Jepsen: MariaDB Galera Cluster by Aphyr 1821 days ago.
	Jepsen: Percona XtraDB Cluster by Aphyr 1821 days ago.
	Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item by Daniel Abadi 1858 days ago.
	Computational techniques in Knossos by Aphyr 2367 days ago.
	Automating Jepsen by Aphyr 2711 days ago.
	Is Upstart the right way to get college student start-ups funded? by Daniel Abadi 2945 days ago.
	Defending Matt Welsh’s 'Startup University' Post by Daniel Abadi 3079 days ago.
	If all these new DBMS technologies are so scalable, why are Oracle and DB2 still on top of TPC-C? A roadmap to end their dominance. by Daniel Abadi 3120 days ago.
	Replication and the latency-consistency tradeoff by Daniel Abadi 3280 days ago.
	Overview of the Oracle NoSQL Database by Daniel Abadi 3342 days ago.
	Hadoop's tremendous inefficiency on graph data management (and how to avoid it) by Daniel Abadi 3413 days ago.
	Why Sam Madden is wrong about peer review by Daniel Abadi 3483 days ago.
	Why I'm doing a start-up pre-tenure by Daniel Abadi 3534 days ago.
	Why I no longer trust EMC [Update: maybe they are not so bad] by Daniel Abadi 3576 days ago.

Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Tue Dec 1 00:21:35