Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Wed Aug 10 01:05:57

	Reporting DBs Question by /u/Firm-Engineer-9909 1 hour ago.
	Azure Cosmos DB Basics \| Zero to Hero Series - 1 by /u/Successful-Aide3077 4 hours ago.
	Looking for help: Spark serverless, dataproc, Hive from AWS or GCP, Azure, jupyter, others? by /u/Objective-Patient-37 6 hours ago.
	Is there a way to add a partition expiration date in spark? by /u/Objective-Patient-37 6 hours ago.
	Dagster Cloud is now GA by /u/schrockn 6 hours ago.
	Does databricks support writing the data in Iceberg format? by /u/No_Load5126 7 hours ago.
	Do Data Engineers own a Data Catalog? by /u/twadftw10 7 hours ago.
	Ability to write production level Software code as a Data engineer by /u/tea_horse 7 hours ago.
	The Data Chasm: Why are companies struggling with data? by /u/curvature_propulsion 7 hours ago.
	Which resource should I pick to learn DE? by /u/Footypajama 7 hours ago.
	Securing Snowflake with Network Access Policies by /u/wspride 8 hours ago.
	How to navigate as a new hire? by /u/Rustic27 9 hours ago.
	Adding custom rows to Spark DF by /u/PR0K1NG 9 hours ago.
	Would love some advice on my career and how to move forward. by /u/Capable-Jicama2155 11 hours ago.
	What's the easiest way to Deploy Airflow 2.3 in Openshift? by /u/Sudden_Quote5886 12 hours ago.
	ingest json messages from S3 to Redshift by /u/mrcool444 12 hours ago.
	On Clickhouse and the open source modern data stack by /u/howMuchCheeseIs2Much 13 hours ago.
	[New job] Laptop choice by /u/GarrusLeSpartan1 13 hours ago.
	Merge avro files with different schemas in pyspark by /u/RstarPhoneix 13 hours ago.
	Auto combining data from multiply databases into one by /u/KitchenSolid 16 hours ago.
	What tools can I use to do constantly updating choropleth maps in a pipeline? by /u/Naive_Income8036 18 hours ago.
	Practical Projects to Learn Data Engineering On AWS by /u/Kickass_Wizard 18 hours ago.
	Looking for feedback to help me transition from entry DA to entry DE by the end of the year if Mid/Sr DA isn’t happening by /u/VAGuy1998 22 hours ago.
	process millions of events per sec by /u/learningpundit 22 hours ago.
	Want to share my Org’s Custom Jinja-heavy dbt dialect by /u/forward-bat-8326 23 hours ago.
	Top 10 Most Powerful Lessons I Learned by Delivering Data Projects by Hanzala Qureshi yesterday
	Spatial Data Science: Installing GDAL on Windows and MacOS by Sutan yesterday
	Detect Defects in a Data Pipeline Early with Validation and Notifications by Khuyen Tran 2 days ago.
	Useful Lessons And Repeatable Patterns Learned From Data Mesh Implementations At AgileLab - Episode 314 by tmacey 3 days ago.
	Optimize Your Machine Learning Development And Serving With The Open Source Vector Database Milvus - Episode 313 by tmacey 3 days ago.
	An Introduction to Databases for Data Scientists by Niklas Lang 3 days ago.
	The Third Wave CDO Blueprint for Career Success by Rohit Choudhary 4 days ago.
	5 Practical SQL Examples To Master SQL GROUP BY by Suraj Gurav 5 days ago.
	Does the Modern Data Stack Value the “Stack” Over “Data”? by Anna Geller 5 days ago.
	Exploring Query Folding in Power Query by Salvatore Cagliari 5 days ago.
	The Origins, Purpose, and Practice of Data Observability by Kevin Hu, PhD 6 days ago.
	Sentiment Analysis on Google Sheets with VADER using Python and Google Cloud Functions by Trent Y., MBA 6 days ago.
	What "Data Lineage Done Right" Looks Like And How They're Doing It At Manta - Episode 312 by tmacey 9 days ago.
	Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda - Episode 311 by tmacey 9 days ago.
	Writing The Book That Offers A Single Reference For The Fundamentals Of Data Engineering - Episode 310 by tmacey 16 days ago.
	Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster - Episode 309 by tmacey 16 days ago.
	IO in PostgreSQL: Past, Present, Future (Andres Freund) by CMU Database Group 17 days ago.
	Orca: A Modular Query Optimizer Architecture for VMware Greenplum (Venkatesh Raghavan) by CMU Database Group 17 days ago.
	RonDB: A Key-Value Store with SQL Capabilities and LATS Properties (Mikael Ronström) by CMU Database Group 17 days ago.
	RavenDB: Practical Considerations for ACID/MVCC Storage Engines (Oren Eini) by CMU Database Group 17 days ago.
	Velox: An Open-source Unified Execution Engine (Deepak Majeti) by CMU Database Group 18 days ago.
	Stardog Query Optimiser: Architecture and Cardinality Estimations for Graph Queries (Pavel Klinov) by CMU Database Group 18 days ago.
	QuestDB: Fast Open Source Time Series Database (Vlad Ilyushchenko) by CMU Database Group 18 days ago.
	SpiceDB: Flexible Permissions Database for the Internet Era (Jake Moshenko) by CMU Database Group 18 days ago.
	Supercharging A/B Testing at Uber by Sergey Gitlin 19 days ago.
	Making The Total Cost Of Ownership For External Data Manageable With Crux - Episode 308 by tmacey 23 days ago.
	Joe Reis Turns The Tables And Interviews Tobias Macey About The Data Engineering Podcast - Episode 307 by tmacey 23 days ago.
	Charting the Path of Riskified's Data Platform Journey - Episode 306 by tmacey 30 days ago.
	Maintain Your Data Engineers' Sanity By Embracing Automation - Episode 305 by tmacey 30 days ago.
	Uber’s Highly Scalable and Distributed Shuffle as a Service by Mayank Bansal 33 days ago.
	Enabling Offline Inferences at Uber Scale by Neeraj Dhake 55 days ago.
	Presto® on Apache Kafka® At Uber Scale by Yang Yang 117 days ago.
	Securing Kafka® Infrastructure at Uber by Prateek Agarwal 124 days ago.
	Yellowbrick: An Elastic Data Warehouse on Kubernetes (Mark Cusack) by CMU Database Group 126 days ago.
	Design and Implementation of the RelationalAI Knowledge Graph Management System (Martin Bravenboer) by CMU Database Group 126 days ago.
	Open-source Change Data Capture With Debezium (Gunnar Morling) by CMU Database Group 146 days ago.
	ApertureDB: Designing a Purpose-built System for Visual Data and Data Science (Vishakha Gupta) by CMU Database Group 152 days ago.
	HTAP with Azure Cosmos DB: Hybrid Transaction & Analytical Processing (Hari Sudan S) by CMU Database Group 173 days ago.
	Project RADAR: Intelligent Early Fraud Detection System with Humans in the Loop by Sergey Zelvenskiy 189 days ago.
	Cost Efficiency @ Scale in Big Data File Format by Xinli Shang 196 days ago.
	Apache Arrow: High-Performance Columnar Data Framework (Wes McKinney) by CMU Database Group 199 days ago.
	How We Build Firebolt (Benjamin Wagner) by CMU Database Group 231 days ago.
	Improving HDFS I/O Utilization for Efficiency by Kevin Cheng 300 days ago.
	Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot by Jacob Tsafatinos 320 days ago.
	YAML Generator for Funnel YAML Files: Streamlining the Mobile Data Workflow Process by Jack Ning 327 days ago.
	Streaming Real-Time Analytics with Redis, AWS Fargate, and Dash Framework by Piyush Choudhary 341 days ago.
	Enabling Seamless Kafka Async Queuing with Consumer Proxy by Yang Yang 343 days ago.
	How Data Shapes the Uber Rider App by Divya Babu Ravichandran 348 days ago.
	Building Scalable Streaming Pipelines for Near Real-Time Features by Feng Xu 350 days ago.
	Efficiently Managing the Supply and Demand on Uber’s Big Data Platform by Zheng Shao 361 days ago.
	Cost-Efficient Open Source Big Data Platform at Uber by Zheng Shao 363 days ago.
	Challenges and Opportunities to Dramatically Reduce the Cost of Uber’s Big Data by Zheng Shao 365 days ago.
	How Uber Achieves Operational Excellence in the Data Quality Experience by Ying Zou 369 days ago.
	Containerizing Apache Hadoop Infrastructure at Uber by Mithun Mathew 383 days ago.
	Analyzing Customer Issues to Improve User Experience by Nimesh Agarwal 389 days ago.
	Automating Merchant Live Monitoring with Real-Time Analytics: Charon by Marco Vita 467 days ago.
	My thoughts on the data mesh by Daniel Abadi 502 days ago.
	Uber’s Journey Toward Better Data Culture From First Principles by Krishna Puttaswamy 511 days ago.
	Turning Metadata Into Insights with Databook by Sunheng Taing 637 days ago.
	Operating Apache Pinot @ Uber Scale by Yupeng Fu 658 days ago.
	Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning by Steffon Davis 798 days ago.
	Monitoring Data Quality at Scale with Statistical Modeling by Ye Henry Li 824 days ago.
	Building a Backtesting Service to Measure Model Performance at Uber-scale by Sam Xiao 908 days ago.
	Women in Data Science at Uber: Moving the World With Data in 2020—and Beyond by Emily Bailey 924 days ago.
	Designing a Production-Ready Kappa Architecture for Timely Data Stream Processing by Amey Chaugule 929 days ago.
	Engineering SQL Support on Apache Pinot at Uber by Haibo Wang 937 days ago.
	Uber Visualization Highlights: Displaying City Street Speed Clusters with SpeedsUp by Bryant Luong 950 days ago.
	It's time to rethink how we share data on the Web by Daniel Abadi 965 days ago.
	Uber’s Data Platform in 2019: Transforming Information to Intelligence by Nikhil Joshi 966 days ago.
	Productionizing Distributed XGBoost to Train Deep Tree Models with Large Data Sets at Uber by Joseph Wang 973 days ago.
	Introducing SLOG: Cheating the low-latency vs. strict serializability tradeoff by Daniel Abadi 978 days ago.
	The dangers of conditional consistency guarantees by Daniel Abadi 978 days ago.
	Introduction to Transaction Isolation Levels by Daniel Abadi 978 days ago.
	Correctness Anomalies Under Serializable Isolation by Daniel Abadi 978 days ago.
	Overview of Consistency Levels in Database Systems by Daniel Abadi 978 days ago.
	An explanation of the difference between Isolation levels vs. Consistency levels by Daniel Abadi 978 days ago.
	Partitioned consensus and its impact on Spanner’s latency by Daniel Abadi 979 days ago.
	It’s Time to Move on from Two Phase Commit by Daniel Abadi 979 days ago.
	NewSQL database systems are failing to guarantee consistency, and I blame Spanner by Daniel Abadi 979 days ago.
	Uber Visualization Highlights: How Urban Symphony Adds an Audio Dimension to Visualization by Jon Sadka 993 days ago.
	Taking City Visualization into the Third Dimension with Point Clouds, 3D Tiles, and deck.gl by Ib Green 1015 days ago.
	Building a Better Big Data Architecture: Meet Uber’s Presto Team by Wayne Cunningham 1035 days ago.
	Science at Uber: Making a Real-world Impact with Data Science by Molly Vorwerck 1085 days ago.
	Less is More: Engineering Data Warehouse Efficiency with Minimalist Design by Ritesh Agrawal 1091 days ago.
	Science at Uber: Powering Uber’s Ridesharing Technologies Through Mapping by Molly Vorwerck 1099 days ago.
	Science at Uber: Bringing Research to the Roads by Molly Vorwerck 1106 days ago.
	An analysis of the strengths and weaknesses of Apache Arrow by Daniel Abadi 1596 days ago.
	Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? by Daniel Abadi 1743 days ago.
	Hazelcast and the Mythical PA/EC System by Daniel Abadi 1766 days ago.
	Scala Days 2017 - Jepsen Keynote by Aphyr 1924 days ago.
	Distributed consistency at scale: Spanner vs. Calvin by Daniel Abadi 1951 days ago.
	Jepsen: CockroachDB beta-20160829 by Aphyr 2000 days ago.
	Jepsen: MongoDB 3.4.0-rc3 by Aphyr 2009 days ago.
	IEEE Computer issue on the CAP Theorem by Daniel Abadi 2185 days ago.
	Jepsen: VoltDB 6.3 by Aphyr 2219 days ago.
	Jepsen: Crate 0.54.9 version divergence by Aphyr 2233 days ago.
	Jepsen: RethinkDB 2.2.3 reconfiguration by Aphyr 2390 days ago.
	Jepsen: RethinkDB 2.1.5 by Aphyr 2409 days ago.
	Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item by Daniel Abadi 2475 days ago.
	Jepsen: Percona XtraDB Cluster by Aphyr 2531 days ago.
	Jepsen: MariaDB Galera Cluster by Aphyr 2534 days ago.
	Jepsen: Chronos by Aphyr 2555 days ago.
	Jepsen: Aerospike by Aphyr 2654 days ago.
	Jepsen: Elasticsearch 1.5.0 by Aphyr 2661 days ago.
	Jepsen: MongoDB stale reads by Aphyr 2668 days ago.
	Jepsen: Elasticsearch by Aphyr 2977 days ago.
	Jepsen: etcd and Consul by Aphyr 2983 days ago.
	Jepsen: RabbitMQ by Aphyr 2986 days ago.
	Computational techniques in Knossos by Aphyr 3005 days ago.
	Strong consistency models by Aphyr 3008 days ago.
	Jepsen: Redis redux by Aphyr 3163 days ago.
	Jepsen: Strangeloop Hangout by Aphyr 3237 days ago.
	Jepsen: RICON East talk by Aphyr 3237 days ago.
	Jepsen: Cassandra by Aphyr 3240 days ago.
	Jepsen: Kafka by Aphyr 3241 days ago.
	Jepsen: NuoDB by Aphyr 3242 days ago.
	Jepsen: Zookeeper by Aphyr 3242 days ago.
	Automating Jepsen by Aphyr 3328 days ago.
	The network is reliable by Aphyr 3355 days ago.
	Asynchronous replication with failover by Aphyr 3367 days ago.
	Jepsen: final thoughts by Aphyr 3368 days ago.
	Jepsen: Riak by Aphyr 3369 days ago.
	Jepsen: MongoDB by Aphyr 3369 days ago.
	Jepsen: On the perils of network partitions by Aphyr 3370 days ago.
	Jepsen: Redis by Aphyr 3370 days ago.
	Jepsen: Postgres by Aphyr 3370 days ago.
	Is Upstart the right way to get college student start-ups funded? by Daniel Abadi 3562 days ago.
	Defending Matt Welsh’s 'Startup University' Post by Daniel Abadi 3696 days ago.
	If all these new DBMS technologies are so scalable, why are Oracle and DB2 still on top of TPC-C? A roadmap to end their dominance. by Daniel Abadi 3737 days ago.
	Replication and the latency-consistency tradeoff by Daniel Abadi 3897 days ago.
	Overview of the Oracle NoSQL Database by Daniel Abadi 3959 days ago.
	Hadoop's tremendous inefficiency on graph data management (and how to avoid it) by Daniel Abadi 4030 days ago.
	Why Sam Madden is wrong about peer review by Daniel Abadi 4100 days ago.
	Why I'm doing a start-up pre-tenure by Daniel Abadi 4151 days ago.

Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Wed Aug 10 01:05:57