Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Sat Jan 16 12:27:58

	I don’t know what to do... by /u/thewizarddan 10 hours ago.
	What options do I have for career growth. by /u/bob7913 16 hours ago.
	Better alternatives to Zeppelin? by /u/ColdPorridge 18 hours ago.
	Catboost, too large to deploy to AWS lambda server, only need a small part of the catboost functionality (predict) by /u/aspiringdatascience 23 hours ago.
	Real life data streaming by /u/Mumo2020 23 hours ago.
	The Journey Towards Metric Standardization by Xiaodong Wang 3 days ago.
	Enabling Version Controlled Data Collaboration With TerminusDB - Episode 167 by tmacey 4 days ago.
	TiDB: On the Long Journey of HTAP (Shawn Ma, PingCAP) by CMU Database Group 8 days ago.
	Bringing Feature Stores and MLOps to the Enterprise At Tecton - Episode 166 by tmacey 11 days ago.
	Off The Shelf Data Governance With Satori - Episode 165 by tmacey 18 days ago.
	Refactoring Query Processing in MySQL (Norvald H. Ryeng, Oracle) by CMU Database Group 21 days ago.
	The Cascades Framework for Query Optimization at Microsoft (Nico Bruno + Cesar Galindo-Legaria) by CMU Database Group 25 days ago.
	Low Friction Data Governance With Immuta - Episode 164 by tmacey 25 days ago.
	Disaster Recovery for Multi-Region Kafka at Uber by Yupeng Fu 25 days ago.
	Uber’s Real-Time Push Platform by Uday Kiran Medisetty 28 days ago.
	Building A Self Service Data Platform For Alternative Data Analytics At YipitData - Episode 163 by tmacey 32 days ago.
	No Code Workflow Orchestrator for Building Batch & Streaming Pipelines at Scale by Sriharsha Chintalapani 35 days ago.
	Proven Patterns For Building Successful Data Teams - Episode 162 by tmacey 39 days ago.
	Streaming Data Integration Without The Code at Equalum - Episode 161 by tmacey 46 days ago.
	ksqlDB: A Stream-Relational Database System (Matthias J. Sax, Confluent) by CMU Database Group 47 days ago.
	Fauna: Lessons Learned Building a Real World, Calvin-based System (Matt Freels, Fauna) by CMU Database Group 47 days ago.
	Horovod v0.21: Optimizing Network Utilization with Local Gradient Aggregation and Grouped Allreduce by Travis Addair 52 days ago.
	Keeping A Bigeye On The Data Quality Market - Episode 160 by tmacey 53 days ago.
	A Deep Dive into Spark SQL's Catalyst Optimizer (Cheng Lian + Maryann Xue, DataBricks) by CMU Database Group 56 days ago.
	Datometry Hyper-Q: Virtualizing the World's Enterprise Data Warehouses (Lyublena Antova, Datometry) by CMU Database Group 57 days ago.
	Self Service Data Management From Ingest To Insights With Isima - Episode 159 by tmacey 60 days ago.
	EraDB: Designing Systems for Cardinality and Dimensionality (Todd Persen, EraDB) by CMU Database Group 65 days ago.
	Turning Metadata Into Insights with Databook by Sunheng Taing 66 days ago.
	Query Optimization at Snowflake (Jiaqi Yan, SnowflakeDB) by CMU Database Group 67 days ago.
	Building A Cost Effective Data Catalog With Tree Schema - Episode 158 by tmacey 67 days ago.
	FoundationDB or: How I Learned to Stop Worrying and Trust the Database (Markus Pilman, Snowflake) by CMU Database Group 71 days ago.
	Meet the 2020 Safety Engineering Interns: COVID Edition by Safety Engineering Interns 78 days ago.
	Operating Apache Pinot @ Uber Scale by Yupeng Fu 87 days ago.
	Building from the Baltics: Meet the Uber Engineering Team in Vilnius, Lithuania by Ieva Sapokaite 93 days ago.
	Apache Arrow Flight: Accelerating Columnar Dataset Transport (Wes McKinney, Ursa Labs) by CMU Database Group 95 days ago.
	Ludwig v0.3 Introduces Hyperparameter Optimization, Transformers and TensorFlow 2 support by Travis Addair 101 days ago.
	Revolutionizing Money Movements at Scale with Strong Data Consistency by Aakriti Singla 105 days ago.
	CockroachDB's Query Optimizer (Rebecca Taft, Cockroach Labs) by CMU Database Group 107 days ago.
	PlanetScale: Query Planning for a Sharded System like Vitess (Sugu Sougoumarane) by CMU Database Group 112 days ago.
	Spearheading Open Source: A Conversation with Jim Jagielski, Staff Technical Program Manager with the Uber Open Source Program Office by Uber Open Source Office 113 days ago.
	ScyllaDB: No-Compromise Performance (Avi Kivity) by CMU Database Group 139 days ago.
	TerminusDB: Building a Native Revision Control DB from Scratch (Gavin Mendel-Gleason) by CMU Database Group 139 days ago.
	Designing Edge Gateway, Uber’s API Lifecycle Management Platform by Madan Thangavelu 150 days ago.
	Standing for Safety: Meet the Uber Sao Paulo Tech Team by Camila Carvalho 171 days ago.
	Introducing Domain-Oriented Microservice Architecture by Adam Gluck 176 days ago.
	Engineering Failover Handling in Uber’s Mobile Networking Infrastructure by Sivabalan Narayanan 185 days ago.
	Fiber: Distributed Computing for AI Made Simple by Jiale Zhi 199 days ago.
	Editing Massive Geospatial Data Sets with nebula.gl by Georgios Karnas 206 days ago.
	Profiles in Coding: Diana Yanakiev, Uber ATG, Pittsburgh by Bea Schuster 213 days ago.
	Jepsen: On the perils of network partitions by Aphyr 217 days ago.
	Building a Large-scale Transactional Data Lake at Uber Using Apache Hudi by Nishith Agarwal 220 days ago.
	Introducing Neuropod, Uber ATG’s Open Source Deep Learning Inference Engine by Vivek Panyam 221 days ago.
	Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning by Steffon Davis 227 days ago.
	Meta-Graph: Few-Shot Link Prediction Using Meta-Learning by Ankit Jain 231 days ago.
	Profiles in Coding: Christabelle Bosson, Uber Elevate, San Francisco by Bea Schuster 232 days ago.
	Developing the Next Generation of Coders with the Dev/Mission Uber Coding Fellowship by Kurtis Nusbaum 238 days ago.
	Introducing Athenadriver: An Open Source Amazon Athena Database Driver for Go by Henry Fuheng Wu 240 days ago.
	Building Uber’s Go Monorepo with Bazel by Zhongpeng Lin 246 days ago.
	Announcing a New Framework for Designing Optimal Experiments with Pyro by Adam Foster 248 days ago.
	Monitoring Data Quality at Scale with Statistical Modeling by Ye Henry Li 253 days ago.
	Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions by Rui Wang 254 days ago.
	Introducing Piranha: An Open Source Tool to Automatically Delete Stale Code by Murali Krishna Ramanathan 304 days ago.
	Fostering a Culture of Sponsorship: Introducing Uber’s Engineering and Sponsorship Development Program by Molly Vorwerck 309 days ago.
	Why We Leverage Multi-tenancy in Uber’s Microservice Architecture by Amit Gud 310 days ago.
	Celebrating International Women’s Day: Meet the Women Building Uber’s Global Platform by Molly Vorwerck 312 days ago.
	Introducing Uber’s Open Source Principles by Brian Hsieh 316 days ago.
	Under the Hood of Uber ATG’s Machine Learning Infrastructure and Versioning Control Platform for Self-Driving Vehicles by Yu Guo 317 days ago.
	Profiles in Coding: Rick Boone, Core Infrastructure, San Francisco by Wayne Cunningham 323 days ago.
	Tricks of the Trade: Tuning JVM Memory for Large-scale Services by Xinli Shang 324 days ago.
	Building the Future of Mobility from the Pacific Northwest: Meet the Uber Seattle Tech Team by Kara Ortbal 332 days ago.
	Building a Backtesting Service to Measure Model Performance at Uber-scale by Sam Xiao 337 days ago.
	It's time to rethink how we share data on the Web by Daniel Abadi 394 days ago.
	Introducing SLOG: Cheating the low-latency vs. strict serializability tradeoff by Daniel Abadi 407 days ago.
	The dangers of conditional consistency guarantees by Daniel Abadi 407 days ago.
	Introduction to Transaction Isolation Levels by Daniel Abadi 407 days ago.
	Correctness Anomalies Under Serializable Isolation by Daniel Abadi 407 days ago.
	Overview of Consistency Levels in Database Systems by Daniel Abadi 407 days ago.
	An explanation of the difference between Isolation levels vs. Consistency levels by Daniel Abadi 407 days ago.
	Partitioned consensus and its impact on Spanner’s latency by Daniel Abadi 408 days ago.
	It’s Time to Move on from Two Phase Commit by Daniel Abadi 408 days ago.
	NewSQL database systems are failing to guarantee consistency, and I blame Spanner by Daniel Abadi 408 days ago.
	Jepsen: Zookeeper by Aphyr 542 days ago.
	Strong consistency models by Aphyr 875 days ago.
	Jepsen: Aerospike by Aphyr 967 days ago.
	An analysis of the strengths and weaknesses of Apache Arrow by Daniel Abadi 1025 days ago.
	Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? by Daniel Abadi 1172 days ago.
	Hazelcast and the Mythical PA/EC System by Daniel Abadi 1195 days ago.
	Jepsen: MongoDB stale reads by Aphyr 1235 days ago.
	Scala Days 2017 - Jepsen Keynote by Aphyr 1353 days ago.
	Distributed consistency at scale: Spanner vs. Calvin by Daniel Abadi 1380 days ago.
	Jepsen: CockroachDB beta-20160829 by Aphyr 1429 days ago.
	Jepsen: MongoDB by Aphyr 1437 days ago.
	Jepsen: MongoDB 3.4.0-rc3 by Aphyr 1438 days ago.
	Jepsen: RethinkDB 2.2.3 reconfiguration by Aphyr 1502 days ago.
	The network is reliable by Aphyr 1523 days ago.
	IEEE Computer issue on the CAP Theorem by Daniel Abadi 1615 days ago.
	Jepsen: Crate 0.54.9 version divergence by Aphyr 1628 days ago.
	Jepsen: VoltDB 6.3 by Aphyr 1647 days ago.
	Jepsen: Cassandra by Aphyr 1669 days ago.
	Asynchronous replication with failover by Aphyr 1711 days ago.
	Jepsen: RethinkDB 2.1.5 by Aphyr 1838 days ago.
	Jepsen: Riak by Aphyr 1868 days ago.
	Jepsen: final thoughts by Aphyr 1868 days ago.
	Jepsen: NuoDB by Aphyr 1868 days ago.
	Jepsen: Kafka by Aphyr 1868 days ago.
	Jepsen: RICON East talk by Aphyr 1868 days ago.
	Jepsen: Strangeloop Hangout by Aphyr 1868 days ago.
	Jepsen: Redis redux by Aphyr 1868 days ago.
	Jepsen: RabbitMQ by Aphyr 1868 days ago.
	Jepsen: etcd and Consul by Aphyr 1868 days ago.
	Jepsen: Elasticsearch by Aphyr 1868 days ago.
	Jepsen: Elasticsearch 1.5.0 by Aphyr 1868 days ago.
	Jepsen: Chronos by Aphyr 1868 days ago.
	Jepsen: MariaDB Galera Cluster by Aphyr 1868 days ago.
	Jepsen: Percona XtraDB Cluster by Aphyr 1868 days ago.
	Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item by Daniel Abadi 1904 days ago.
	Computational techniques in Knossos by Aphyr 2414 days ago.
	Automating Jepsen by Aphyr 2758 days ago.
	Is Upstart the right way to get college student start-ups funded? by Daniel Abadi 2991 days ago.
	Defending Matt Welsh’s 'Startup University' Post by Daniel Abadi 3125 days ago.
	If all these new DBMS technologies are so scalable, why are Oracle and DB2 still on top of TPC-C? A roadmap to end their dominance. by Daniel Abadi 3166 days ago.
	Replication and the latency-consistency tradeoff by Daniel Abadi 3326 days ago.
	Overview of the Oracle NoSQL Database by Daniel Abadi 3388 days ago.
	Hadoop's tremendous inefficiency on graph data management (and how to avoid it) by Daniel Abadi 3459 days ago.
	Why Sam Madden is wrong about peer review by Daniel Abadi 3529 days ago.
	Why I'm doing a start-up pre-tenure by Daniel Abadi 3580 days ago.
	Why I no longer trust EMC [Update: maybe they are not so bad] by Daniel Abadi 3622 days ago.

Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Sat Jan 16 12:27:58