Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Wed Apr 14 00:39:08

	Hows the job market in Australia? by /u/Aspiring_DE 52 minutes ago.
	Amazon Vendor/Seller API by /u/closetedswiftie 2 hours ago.
	Apache Spark vs Ray Comparison by /u/mgalarny 4 hours ago.
	The deck we used to raise our Seed round for our open-source EL(T) platform by /u/jeanlaf 4 hours ago.
	A table with user id's in databricks spark? by /u/bhargavn07 6 hours ago.
	Breaking into data engineering by /u/hunterna23 7 hours ago.
	Examples of professional data pipelines? by /u/ambiguousjellyfish 8 hours ago.
	What steps can I take to become a data engineer by /u/counttoukou95 10 hours ago.
	Heterogeneous graph analysis tools suggestions by /u/theweirdinstruction 13 hours ago.
	DataCamp Data Engineer with Python course by /u/etx369 13 hours ago.
	Comparing Hudi, Iceberg and Delta Lake by /u/Bergamask_85 16 hours ago.
	Exploring The Expanding Landscape Of Data Professions with Josh Benamram of Databand - Episode 180 by tmacey 22 hours ago.
	Flipr: Making Changes Quickly and Safely at Scale by Andy Maule yesterday
	Query Processing in Google BigQuery (Hossein Ahmadi + Aleksandras Surna) by CMU Database Group 6 days ago.
	Put Your Whole Data Team On The Same Page With Atlan - Episode 179 by tmacey 7 days ago.
	FASTER: Efficient State Management for the Modern Edge-Cloud (Badrish Chandramouli) by CMU Database Group 8 days ago.
	Data Quality Management For The Whole Team With Soda Data - Episode 178 by tmacey 14 days ago.
	NoisePage: The Self-Driving Database Management System (Lin Ma) by CMU Database Group 19 days ago.
	My thoughts on the data mesh by Daniel Abadi 19 days ago.
	Real World Change Data Capture At Datacoral - Episode 177 by tmacey 22 days ago.
	HarperDB’s Data Storage Journey: From File System to LMDB (Kyle Bernhardy) by CMU Database Group 24 days ago.
	Celebration of Mike Stonebraker #1 - Introduction (Andy Pavlo) by CMU Database Group 24 days ago.
	Celebration of Mike Stonebraker #6 - MIT Impact (Sam Madden) by CMU Database Group 26 days ago.
	Celebration of Mike Stonebraker #8 - Students (Magda Balazinska, Andy Pavlo) by CMU Database Group 26 days ago.
	Celebration of Mike Stonebraker #3 - Ingres / Berkeley Panel (Allman, Held, Selinger, Carey) by CMU Database Group 26 days ago.
	Celebration of Mike Stonebraker #10 - Closing Remarks (Mike Stonebraker) by CMU Database Group 27 days ago.
	Novel Design Choices in Apache CouchDB (Adam Kocoloski) by CMU Database Group 27 days ago.
	Uber’s Journey Toward Better Data Culture From First Principles by Krishna Puttaswamy 28 days ago.
	Managing The DoorDash Data Platform - Episode 176 by tmacey 28 days ago.
	Celebration of Mike Stonebraker #5 - Streaming Databases (Stan Zdonik) by CMU Database Group 29 days ago.
	Celebration of Mike Stonebraker #4 - Postgres & Beyond (Jhingran, Olson, Hellerstein, Hong, Seltzer) by CMU Database Group 31 days ago.
	Celebration of Mike Stonebraker #2 - 44 Years of Mike (David DeWitt) by CMU Database Group 31 days ago.
	Celebration of Mike Stonebraker #7 - Vertica & Column Stores (Andy Palmer) by CMU Database Group 31 days ago.
	Celebration of Mike Stonebraker #9 - Open Mic + Messages by CMU Database Group 31 days ago.
	Navigating to the Technical Program Management and Learning Team 32 days ago.
	Leave Your Data Where It Is And Automate Feature Extraction With Molecula - Episode 175 by tmacey 36 days ago.
	Elastic Deep Learning with Horovod on Ray by Travis Addair 36 days ago.
	Applying Machine Learning in Internal Audit with Sparsely Labeled Data by Jesse He 42 days ago.
	Bridging The Gap Between Machine Learning And Operations At Iguazio - Episode 174 by tmacey 42 days ago.
	How Uber Deals with Large iOS App Size by Milind Chabbi 46 days ago.
	Evolving Schemaless into a Distributed SQL Database by Ovais Tariq 49 days ago.
	Self Service Open Source Data Integration With AirByte - Episode 173 by tmacey 49 days ago.
	Fast and Reliable Schema-Agnostic Log Analytics Platform by Chao Wang 53 days ago.
	Building The Foundations For Data Driven Businesses at 5xData - Episode 172 by tmacey 56 days ago.
	How Shopify Is Building Their Production Data Warehouse Using DBT - Episode 171 by tmacey 63 days ago.
	Uber’s Real-time Data Intelligence Platform At Scale: Improving Gairos Scalability/Reliability by Gang Zhao 84 days ago.
	The Journey Towards Metric Standardization by Xiaodong Wang 91 days ago.
	Disaster Recovery for Multi-Region Kafka at Uber by Yupeng Fu 113 days ago.
	Uber’s Real-Time Push Platform by Uday Kiran Medisetty 116 days ago.
	No Code Workflow Orchestrator for Building Batch & Streaming Pipelines at Scale by Sriharsha Chintalapani 123 days ago.
	Horovod v0.21: Optimizing Network Utilization with Local Gradient Aggregation and Grouped Allreduce by Travis Addair 140 days ago.
	Turning Metadata Into Insights with Databook by Sunheng Taing 154 days ago.
	Meet the 2020 Safety Engineering Interns: COVID Edition by Safety Engineering Interns 166 days ago.
	Operating Apache Pinot @ Uber Scale by Yupeng Fu 175 days ago.
	Building from the Baltics: Meet the Uber Engineering Team in Vilnius, Lithuania by Ieva Sapokaite 180 days ago.
	Ludwig v0.3 Introduces Hyperparameter Optimization, Transformers and TensorFlow 2 support by Travis Addair 189 days ago.
	Revolutionizing Money Movements at Scale with Strong Data Consistency by Aakriti Singla 193 days ago.
	Spearheading Open Source: A Conversation with Jim Jagielski, Staff Technical Program Manager with the Uber Open Source Program Office by Uber Open Source Office 201 days ago.
	Designing Edge Gateway, Uber’s API Lifecycle Management Platform by Madan Thangavelu 238 days ago.
	Standing for Safety: Meet the Uber Sao Paulo Tech Team by Camila Carvalho 259 days ago.
	Introducing Domain-Oriented Microservice Architecture by Adam Gluck 264 days ago.
	Engineering Failover Handling in Uber’s Mobile Networking Infrastructure by Sivabalan Narayanan 273 days ago.
	Fiber: Distributed Computing for AI Made Simple by Jiale Zhi 287 days ago.
	Editing Massive Geospatial Data Sets with nebula.gl by Georgios Karnas 294 days ago.
	Profiles in Coding: Diana Yanakiev, Uber ATG, Pittsburgh by Bea Schuster 301 days ago.
	Building a Large-scale Transactional Data Lake at Uber Using Apache Hudi by Nishith Agarwal 308 days ago.
	Introducing Neuropod, Uber ATG’s Open Source Deep Learning Inference Engine by Vivek Panyam 309 days ago.
	Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning by Steffon Davis 315 days ago.
	Meta-Graph: Few-Shot Link Prediction Using Meta-Learning by Ankit Jain 319 days ago.
	Profiles in Coding: Christabelle Bosson, Uber Elevate, San Francisco by Bea Schuster 320 days ago.
	Developing the Next Generation of Coders with the Dev/Mission Uber Coding Fellowship by Kurtis Nusbaum 326 days ago.
	Introducing Athenadriver: An Open Source Amazon Athena Database Driver for Go by Henry Fuheng Wu 328 days ago.
	Building Uber’s Go Monorepo with Bazel by Zhongpeng Lin 334 days ago.
	Announcing a New Framework for Designing Optimal Experiments with Pyro by Adam Foster 336 days ago.
	Monitoring Data Quality at Scale with Statistical Modeling by Ye Henry Li 341 days ago.
	Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions by Rui Wang 342 days ago.
	Introducing Piranha: An Open Source Tool to Automatically Delete Stale Code by Murali Krishna Ramanathan 392 days ago.
	It's time to rethink how we share data on the Web by Daniel Abadi 482 days ago.
	Introducing SLOG: Cheating the low-latency vs. strict serializability tradeoff by Daniel Abadi 495 days ago.
	The dangers of conditional consistency guarantees by Daniel Abadi 495 days ago.
	Introduction to Transaction Isolation Levels by Daniel Abadi 495 days ago.
	Correctness Anomalies Under Serializable Isolation by Daniel Abadi 495 days ago.
	Overview of Consistency Levels in Database Systems by Daniel Abadi 495 days ago.
	An explanation of the difference between Isolation levels vs. Consistency levels by Daniel Abadi 495 days ago.
	Partitioned consensus and its impact on Spanner’s latency by Daniel Abadi 496 days ago.
	It’s Time to Move on from Two Phase Commit by Daniel Abadi 496 days ago.
	NewSQL database systems are failing to guarantee consistency, and I blame Spanner by Daniel Abadi 496 days ago.
	An analysis of the strengths and weaknesses of Apache Arrow by Daniel Abadi 1113 days ago.
	Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? by Daniel Abadi 1260 days ago.
	Hazelcast and the Mythical PA/EC System by Daniel Abadi 1283 days ago.
	Scala Days 2017 - Jepsen Keynote by Aphyr 1441 days ago.
	Distributed consistency at scale: Spanner vs. Calvin by Daniel Abadi 1468 days ago.
	Jepsen: CockroachDB beta-20160829 by Aphyr 1517 days ago.
	Jepsen: MongoDB 3.4.0-rc3 by Aphyr 1526 days ago.
	IEEE Computer issue on the CAP Theorem by Daniel Abadi 1702 days ago.
	Jepsen: VoltDB 6.3 by Aphyr 1736 days ago.
	Jepsen: Crate 0.54.9 version divergence by Aphyr 1750 days ago.
	Jepsen: RethinkDB 2.2.3 reconfiguration by Aphyr 1907 days ago.
	Jepsen: RethinkDB 2.1.5 by Aphyr 1926 days ago.
	Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item by Daniel Abadi 1992 days ago.
	Jepsen: Percona XtraDB Cluster by Aphyr 2048 days ago.
	Jepsen: MariaDB Galera Cluster by Aphyr 2051 days ago.
	Jepsen: Chronos by Aphyr 2072 days ago.
	Jepsen: Aerospike by Aphyr 2171 days ago.
	Jepsen: Elasticsearch 1.5.0 by Aphyr 2178 days ago.
	Jepsen: MongoDB stale reads by Aphyr 2185 days ago.
	Jepsen: Elasticsearch by Aphyr 2494 days ago.
	Jepsen: etcd and Consul by Aphyr 2500 days ago.
	Jepsen: RabbitMQ by Aphyr 2503 days ago.
	Computational techniques in Knossos by Aphyr 2522 days ago.
	Strong consistency models by Aphyr 2525 days ago.
	Jepsen: Redis redux by Aphyr 2680 days ago.
	Jepsen: Strangeloop Hangout by Aphyr 2754 days ago.
	Jepsen: RICON East talk by Aphyr 2754 days ago.
	Jepsen: Cassandra by Aphyr 2757 days ago.
	Jepsen: Kafka by Aphyr 2758 days ago.
	Jepsen: NuoDB by Aphyr 2759 days ago.
	Jepsen: Zookeeper by Aphyr 2759 days ago.
	Automating Jepsen by Aphyr 2845 days ago.
	The network is reliable by Aphyr 2872 days ago.
	Asynchronous replication with failover by Aphyr 2884 days ago.
	Jepsen: final thoughts by Aphyr 2885 days ago.
	Jepsen: Riak by Aphyr 2886 days ago.
	Jepsen: MongoDB by Aphyr 2886 days ago.
	Jepsen: On the perils of network partitions by Aphyr 2887 days ago.
	Jepsen: Redis by Aphyr 2887 days ago.
	Jepsen: Postgres by Aphyr 2887 days ago.
	Is Upstart the right way to get college student start-ups funded? by Daniel Abadi 3079 days ago.
	Defending Matt Welsh’s 'Startup University' Post by Daniel Abadi 3213 days ago.
	If all these new DBMS technologies are so scalable, why are Oracle and DB2 still on top of TPC-C? A roadmap to end their dominance. by Daniel Abadi 3254 days ago.
	Replication and the latency-consistency tradeoff by Daniel Abadi 3414 days ago.
	Overview of the Oracle NoSQL Database by Daniel Abadi 3476 days ago.
	Hadoop's tremendous inefficiency on graph data management (and how to avoid it) by Daniel Abadi 3547 days ago.
	Why Sam Madden is wrong about peer review by Daniel Abadi 3617 days ago.
	Why I'm doing a start-up pre-tenure by Daniel Abadi 3668 days ago.

Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Wed Apr 14 00:39:08