Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Sat Mar 6 00:21:17

	Space-time trade off and your Snowflake compute cost by /u/ssb61 1 hour ago.
	How can you ensure you don't have duplicate/missing values in a streaming application that writes to a non relational DB? by /u/IGuessYourSubreddits 2 hours ago.
	ETL by /u/anurag_bhoga 4 hours ago.
	Best resource/book to learn difference between NoSQL vs relational(SQL) databases? by /u/crpablo 5 hours ago.
	I need your advise :) by /u/winters2o 6 hours ago.
	Single-node and distributed Deep Learning on Databricks by /u/squareape 6 hours ago.
	Courses to help transition from Vertica to MS SQL by /u/Glibergoo_bop 8 hours ago.
	Introduction to Redshift using Pagila Sample Dataset Including ETL from Postgres using AWS Glue by /u/amcquistan 8 hours ago.
	Do you guys also get contacted directly by recruiters for DE roles? by /u/1337codethrow 17 hours ago.
	Building Real-time Data Architectures to Foster Innovation by /u/ssb61 17 hours ago.
	PySpark question: RDD Api vs SparkSql Api? by /u/1337codethrow 21 hours ago.
	Inside Apache Druid's Storage and Query Engine (Gian Merlino) by CMU Database Group 2 days ago.
	Applying Machine Learning in Internal Audit with Sparsely Labeled Data by Jesse He 3 days ago.
	Bridging The Gap Between Machine Learning And Operations At Iguazio - Episode 174 by tmacey 3 days ago.
	How Uber Deals with Large iOS App Size by Milind Chabbi 7 days ago.
	Citus: Distributed PostgreSQL as an Extension (Microsoft Citus Data) by CMU Database Group 9 days ago.
	Evolving Schemaless into a Distributed SQL Database by Ovais Tariq 10 days ago.
	Self Service Open Source Data Integration With AirByte - Episode 173 by tmacey 10 days ago.
	Fast and Reliable Schema-Agnostic Log Analytics Platform by Chao Wang 14 days ago.
	Star-Tree Index: Space-Time Trade Off in OLAP (Kishore Gopalakrishna, Apache Pinot) by CMU Database Group 14 days ago.
	Building The Foundations For Data Driven Businesses at 5xData - Episode 172 by tmacey 17 days ago.
	Performance Testing at MongoDB (David Daly, MongoDB) by CMU Database Group 23 days ago.
	SLOG: Serializable, Low-latency, Geo-replicated Transactions (Daniel Abadi, University of Maryland) by CMU Database Group 23 days ago.
	How Shopify Is Building Their Production Data Warehouse Using DBT - Episode 171 by tmacey 24 days ago.
	System Observability For The Cloud Native Era With Chronosphere - Episode 170 by tmacey 31 days ago.
	Making It Easier To Stick B2B Data Integration Pipelines Together With Hotglue - Episode 169 by tmacey 38 days ago.
	Uber’s Real-time Data Intelligence Platform At Scale: Improving Gairos Scalability/Reliability by Gang Zhao 45 days ago.
	Using Your Data Warehouse As The Source Of Truth For Customer Data With Hightouch - Episode 168 by tmacey 45 days ago.
	The Journey Towards Metric Standardization by Xiaodong Wang 52 days ago.
	Enabling Version Controlled Data Collaboration With TerminusDB - Episode 167 by tmacey 53 days ago.
	TiDB: On the Long Journey of HTAP (Shawn Ma, PingCAP) by CMU Database Group 57 days ago.
	Bringing Feature Stores and MLOps to the Enterprise At Tecton - Episode 166 by tmacey 59 days ago.
	Off The Shelf Data Governance With Satori - Episode 165 by tmacey 67 days ago.
	Refactoring Query Processing in MySQL (Norvald H. Ryeng, Oracle) by CMU Database Group 70 days ago.
	The Cascades Framework for Query Optimization at Microsoft (Nico Bruno + Cesar Galindo-Legaria) by CMU Database Group 73 days ago.
	Disaster Recovery for Multi-Region Kafka at Uber by Yupeng Fu 74 days ago.
	Uber’s Real-Time Push Platform by Uday Kiran Medisetty 77 days ago.
	No Code Workflow Orchestrator for Building Batch & Streaming Pipelines at Scale by Sriharsha Chintalapani 84 days ago.
	ksqlDB: A Stream-Relational Database System (Matthias J. Sax, Confluent) by CMU Database Group 96 days ago.
	Fauna: Lessons Learned Building a Real World, Calvin-based System (Matt Freels, Fauna) by CMU Database Group 96 days ago.
	Horovod v0.21: Optimizing Network Utilization with Local Gradient Aggregation and Grouped Allreduce by Travis Addair 101 days ago.
	A Deep Dive into Spark SQL's Catalyst Optimizer (Cheng Lian + Maryann Xue, DataBricks) by CMU Database Group 105 days ago.
	Datometry Hyper-Q: Virtualizing the World's Enterprise Data Warehouses (Lyublena Antova, Datometry) by CMU Database Group 106 days ago.
	EraDB: Designing Systems for Cardinality and Dimensionality (Todd Persen, EraDB) by CMU Database Group 114 days ago.
	Turning Metadata Into Insights with Databook by Sunheng Taing 115 days ago.
	FoundationDB or: How I Learned to Stop Worrying and Trust the Database (Markus Pilman, Snowflake) by CMU Database Group 119 days ago.
	Meet the 2020 Safety Engineering Interns: COVID Edition by Safety Engineering Interns 127 days ago.
	Operating Apache Pinot @ Uber Scale by Yupeng Fu 136 days ago.
	Building from the Baltics: Meet the Uber Engineering Team in Vilnius, Lithuania by Ieva Sapokaite 141 days ago.
	Apache Arrow Flight: Accelerating Columnar Dataset Transport (Wes McKinney, Ursa Labs) by CMU Database Group 144 days ago.
	Ludwig v0.3 Introduces Hyperparameter Optimization, Transformers and TensorFlow 2 support by Travis Addair 150 days ago.
	Revolutionizing Money Movements at Scale with Strong Data Consistency by Aakriti Singla 154 days ago.
	Spearheading Open Source: A Conversation with Jim Jagielski, Staff Technical Program Manager with the Uber Open Source Program Office by Uber Open Source Office 162 days ago.
	Designing Edge Gateway, Uber’s API Lifecycle Management Platform by Madan Thangavelu 199 days ago.
	Standing for Safety: Meet the Uber Sao Paulo Tech Team by Camila Carvalho 220 days ago.
	Introducing Domain-Oriented Microservice Architecture by Adam Gluck 225 days ago.
	Engineering Failover Handling in Uber’s Mobile Networking Infrastructure by Sivabalan Narayanan 234 days ago.
	Fiber: Distributed Computing for AI Made Simple by Jiale Zhi 248 days ago.
	Editing Massive Geospatial Data Sets with nebula.gl by Georgios Karnas 255 days ago.
	Profiles in Coding: Diana Yanakiev, Uber ATG, Pittsburgh by Bea Schuster 262 days ago.
	Jepsen: On the perils of network partitions by Aphyr 265 days ago.
	Building a Large-scale Transactional Data Lake at Uber Using Apache Hudi by Nishith Agarwal 269 days ago.
	Introducing Neuropod, Uber ATG’s Open Source Deep Learning Inference Engine by Vivek Panyam 270 days ago.
	Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning by Steffon Davis 276 days ago.
	Meta-Graph: Few-Shot Link Prediction Using Meta-Learning by Ankit Jain 280 days ago.
	Profiles in Coding: Christabelle Bosson, Uber Elevate, San Francisco by Bea Schuster 281 days ago.
	Developing the Next Generation of Coders with the Dev/Mission Uber Coding Fellowship by Kurtis Nusbaum 287 days ago.
	Introducing Athenadriver: An Open Source Amazon Athena Database Driver for Go by Henry Fuheng Wu 289 days ago.
	Building Uber’s Go Monorepo with Bazel by Zhongpeng Lin 295 days ago.
	Announcing a New Framework for Designing Optimal Experiments with Pyro by Adam Foster 297 days ago.
	Monitoring Data Quality at Scale with Statistical Modeling by Ye Henry Li 302 days ago.
	Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions by Rui Wang 303 days ago.
	Introducing Piranha: An Open Source Tool to Automatically Delete Stale Code by Murali Krishna Ramanathan 353 days ago.
	Fostering a Culture of Sponsorship: Introducing Uber’s Engineering and Sponsorship Development Program by Molly Vorwerck 358 days ago.
	Why We Leverage Multi-tenancy in Uber’s Microservice Architecture by Amit Gud 359 days ago.
	Celebrating International Women’s Day: Meet the Women Building Uber’s Global Platform by Molly Vorwerck 361 days ago.
	Introducing Uber’s Open Source Principles by Brian Hsieh 365 days ago.
	It's time to rethink how we share data on the Web by Daniel Abadi 443 days ago.
	Introducing SLOG: Cheating the low-latency vs. strict serializability tradeoff by Daniel Abadi 456 days ago.
	The dangers of conditional consistency guarantees by Daniel Abadi 456 days ago.
	Introduction to Transaction Isolation Levels by Daniel Abadi 456 days ago.
	Correctness Anomalies Under Serializable Isolation by Daniel Abadi 456 days ago.
	Overview of Consistency Levels in Database Systems by Daniel Abadi 456 days ago.
	An explanation of the difference between Isolation levels vs. Consistency levels by Daniel Abadi 456 days ago.
	Partitioned consensus and its impact on Spanner’s latency by Daniel Abadi 457 days ago.
	It’s Time to Move on from Two Phase Commit by Daniel Abadi 457 days ago.
	NewSQL database systems are failing to guarantee consistency, and I blame Spanner by Daniel Abadi 457 days ago.
	Jepsen: Zookeeper by Aphyr 591 days ago.
	Strong consistency models by Aphyr 924 days ago.
	Jepsen: Aerospike by Aphyr 1016 days ago.
	An analysis of the strengths and weaknesses of Apache Arrow by Daniel Abadi 1074 days ago.
	Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? by Daniel Abadi 1221 days ago.
	Hazelcast and the Mythical PA/EC System by Daniel Abadi 1244 days ago.
	Jepsen: MongoDB stale reads by Aphyr 1284 days ago.
	Scala Days 2017 - Jepsen Keynote by Aphyr 1402 days ago.
	Distributed consistency at scale: Spanner vs. Calvin by Daniel Abadi 1429 days ago.
	Jepsen: CockroachDB beta-20160829 by Aphyr 1478 days ago.
	Jepsen: MongoDB by Aphyr 1486 days ago.
	Jepsen: MongoDB 3.4.0-rc3 by Aphyr 1487 days ago.
	Jepsen: RethinkDB 2.2.3 reconfiguration by Aphyr 1551 days ago.
	The network is reliable by Aphyr 1572 days ago.
	IEEE Computer issue on the CAP Theorem by Daniel Abadi 1663 days ago.
	Jepsen: Crate 0.54.9 version divergence by Aphyr 1677 days ago.
	Jepsen: VoltDB 6.3 by Aphyr 1696 days ago.
	Jepsen: Cassandra by Aphyr 1718 days ago.
	Asynchronous replication with failover by Aphyr 1759 days ago.
	Jepsen: RethinkDB 2.1.5 by Aphyr 1887 days ago.
	Jepsen: Riak by Aphyr 1916 days ago.
	Jepsen: final thoughts by Aphyr 1916 days ago.
	Jepsen: NuoDB by Aphyr 1916 days ago.
	Jepsen: Kafka by Aphyr 1916 days ago.
	Jepsen: RICON East talk by Aphyr 1916 days ago.
	Jepsen: Strangeloop Hangout by Aphyr 1916 days ago.
	Jepsen: Redis redux by Aphyr 1916 days ago.
	Jepsen: RabbitMQ by Aphyr 1916 days ago.
	Jepsen: etcd and Consul by Aphyr 1916 days ago.
	Jepsen: Elasticsearch by Aphyr 1916 days ago.
	Jepsen: Elasticsearch 1.5.0 by Aphyr 1916 days ago.
	Jepsen: Chronos by Aphyr 1916 days ago.
	Jepsen: MariaDB Galera Cluster by Aphyr 1916 days ago.
	Jepsen: Percona XtraDB Cluster by Aphyr 1916 days ago.
	Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item by Daniel Abadi 1953 days ago.
	Computational techniques in Knossos by Aphyr 2462 days ago.
	Automating Jepsen by Aphyr 2806 days ago.
	Is Upstart the right way to get college student start-ups funded? by Daniel Abadi 3040 days ago.
	Defending Matt Welsh’s 'Startup University' Post by Daniel Abadi 3174 days ago.
	If all these new DBMS technologies are so scalable, why are Oracle and DB2 still on top of TPC-C? A roadmap to end their dominance. by Daniel Abadi 3215 days ago.
	Replication and the latency-consistency tradeoff by Daniel Abadi 3375 days ago.
	Overview of the Oracle NoSQL Database by Daniel Abadi 3437 days ago.
	Hadoop's tremendous inefficiency on graph data management (and how to avoid it) by Daniel Abadi 3508 days ago.
	Why Sam Madden is wrong about peer review by Daniel Abadi 3578 days ago.
	Why I'm doing a start-up pre-tenure by Daniel Abadi 3629 days ago.
	Why I no longer trust EMC [Update: maybe they are not so bad] by Daniel Abadi 3671 days ago.

Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Sat Mar 6 00:21:17