{"id":3011420,"date":"2026-04-22T10:47:49","date_gmt":"2026-04-22T17:47:49","guid":{"rendered":"urn:uuid:657308ec-ec79-48e3-b97b-c3ac6292e56b"},"modified":"2026-04-22T10:47:49","modified_gmt":"2026-04-22T17:47:49","slug":"perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality","status":"publish","type":"research-post","link":"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/","title":{"rendered":"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality"},"content":{"rendered":"\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:70%\">\t\t<div\n\t\t\tclass=\"wp-block-sie-social-share social-share--raw\"\n\t\t\tdata-aa-modulename=\"social-share\"\n\t\t\tdata-url=\"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/\"\n\t\t\tdata-title=\"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality\"\n\t\t\tdata-heading=\"Share this\"\n\t\t\tdata-bluesky-title=\"\"\n\t\t\tdata-twitter-title=\"\"\n\t\t\tdata-twitter-hashtags=\"\"\n\t\t\tdata-reddit-title=\"\"\n\t\t\tdata-email-subject=\"\"\n\t\t\tdata-email-body=\"\"\n\t\t\t>\n\t\t<\/div>\n\t\n\n<div class=\"post-author\" data-no-of-bylines=\"8\"><ul><li>Yunxiang Zhang<span class=\"sie-style-body-small\">PhD Student, New York University<\/span><\/li><li>Sai Harsha Mupparaju<span class=\"sie-style-body-small\">Master\u2019s Student, New York University<\/span><\/li><li>Kenneth Chen<span class=\"sie-style-body-small\">PhD Student, New York University<\/span><\/li><li>Jenna Kang<span class=\"sie-style-body-small\">PhD Student, New York University<\/span><\/li><li>Xinyu Zhang<span class=\"sie-style-body-small\">Machine Learning Researcher II, Sony Interactive Entertainment<\/span><\/li><li>Maito Omori<span class=\"sie-style-body-small\">Software Engineer II, Sony Interactive Entertainment<\/span><\/li><li>Kazuyuki Arimatsu<span class=\"sie-style-body-small\">Sr. Software Researcher, Sony Interactive Entertainment<\/span><\/li><li>Qi Sun<span class=\"sie-style-body-small\">Associate Professor, New York University<\/span><\/li><\/ul><\/div>\n\n\n<p class=\"sie-paragraph sie-paragraph-083d3844-745f-4760-9515-68a3508d9d47\">Recent breakthroughs in radiance fields, particularly 3D Gaussian Splatting (3DGS), have unlocked real-time, high-quality rendering of complex environments, enabling a wide range of applications. However, the stringent requirements of mixed reality (MR) rendering, such as rapid refresh rates, high-resolution stereo viewing, and constrained computing budgets, remain out of reach for current 3DGS techniques. Nevertheless, the wide field-of-view design of MR displays, which mimics human vision, presents a unique opportunity to exploit human visual system&#8217;s own perceptual limitations to reduce computational overhead while not compromising user-perceived rendering quality.&nbsp;<\/p>\n\n\n\n<p class=\"sie-paragraph sie-paragraph-462ae573-6f28-425d-bad4-5f134d15d6e7\">To this end, we propose a perception-guided, continuous level-of-detail (LOD) framework for 3DGS that maximizes perceived quality under given compute resources. We distill a visual quality metric, which encodes the spatial, temporal, and peripheral characteristics of human visual perception, into a lightweight, gaze-contingent model that predicts and adaptively modulates rendering LOD across the visual field based on each region&#8217;s contributions to perceptual quality. This budget-driven LOD modulation, guided by both scene content and gaze behavior, enables significant computation reduction with minimal loss in perceived quality. To support low-power, untethered MR setups, we design an edge-cloud collaborative rendering framework to partially offload computation to the cloud, further reducing overhead on the edge MR devices. Objective metrics and MR user study evidence that, compared to vanilla and foveated LOD baselines, our method achieves superior trade-offs between computational efficiency and user-perceived visual quality.<\/p>\n\n\n\n<p class=\"sie-paragraph sie-paragraph-27fe01b0-20c2-4a9e-a967-273401dd7ce6\">Access the full research <a class=\"sie-paragraph sie-paragraph-27fe01b0-20c2-4a9e-a967-273401dd7ce6\" href=\"https:\/\/openaccess.thecvf.com\/content\/WACV2026\/html\/Zhang_Perceptually_Guided_3DGS_Streaming_and_Rendering_for_Mixed_Reality_WACV_2026_paper.html\">here<\/a>.<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n\t\t<div class=\"related-posts__container\" data-selection-type=\"latest\" data-post-type=\"research-post\">\n\t\t\t<h2 class=\"related-posts__heading sie-style-h5\">Latest Research Posts<\/h2>\n\t\t\t<div class=\"related-posts related-posts--vertical\">\n\t\t\t\t<article class=\"related-post related-post--research-post\"><div class=\"related-post__content\"><h3 class=\"related-post__title sie-style-body-small-v2\">\n\t\t<a href=\"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/content-adaptive-encoding-for-interactive-game-streaming\/\">\n\t\t\tContent Adaptive Encoding for Interactive Game Streaming\n\t\t<\/a>\n\t<\/h3><\/div><\/article><article class=\"related-post related-post--research-post\"><div class=\"related-post__content\"><h3 class=\"related-post__title sie-style-body-small-v2\">\n\t\t<a href=\"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/vision-language-models-for-quality-assurance\/\">\n\t\t\tVideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance\n\t\t<\/a>\n\t<\/h3><\/div><\/article><article class=\"related-post related-post--research-post\"><div class=\"related-post__content\"><h3 class=\"related-post__title sie-style-body-small-v2\">\n\t\t<a href=\"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/learning-representations-in-video-game-agents\/\">\n\t\t\tLearning Representations in Video Game Agents with Supervised Contrastive Imitation Learning\n\t\t<\/a>\n\t<\/h3><\/div><\/article>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-sie-scroll-to-top\" data-aa-modulename=\"sie-scroll-to-top\"><button class=\"sie-btn sie-btn--action\" type=\"button\"><span>Back to top<\/span><\/button><\/div>\n\n\n\n<p class=\"sie-paragraph sie-paragraph-1a9b971c-265a-4a92-bb46-787167158a72\"><\/p>\n\n\n\n<p class=\"sie-paragraph sie-paragraph-b14e0cb2-eeef-43fb-b063-194b42e4eaeb\"><\/p>\n\n\n\n<p class=\"sie-paragraph sie-paragraph-415dee72-9225-41b9-a1d6-b45decd94d81\"><\/p>\n","protected":false},"author":35,"parent":0,"template":"","byline":[415,416,417,418,419,420,421,422],"research-post-category":[],"class_list":["post-3011420","research-post","type-research-post","status-publish","hentry","post-perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality"],"ab_tests":{},"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.4 (Yoast SEO v27.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality - Sony Interactive Entertainment<\/title>\n<meta name=\"description\" content=\"New York University and Sony Interactive Entertainment research on perception-guided 3D Gaussian Splatting framework for mixed reality\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality\" \/>\n<meta property=\"og:description\" content=\"New York University and Sony Interactive Entertainment research on perception-guided 3D Gaussian Splatting framework for mixed reality\" \/>\n<meta property=\"og:url\" content=\"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/\" \/>\n<meta property=\"og:site_name\" content=\"Sony Interactive Entertainment\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data2\" content=\"Yunxiang Zhang\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/innovation\\\/research-academia\\\/research\\\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\\\/\",\"url\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/innovation\\\/research-academia\\\/research\\\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\\\/\",\"name\":\"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality - Sony Interactive Entertainment\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/#website\"},\"datePublished\":\"2026-04-22T17:47:49+00:00\",\"description\":\"New York University and Sony Interactive Entertainment research on perception-guided 3D Gaussian Splatting framework for mixed reality\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/innovation\\\/research-academia\\\/research\\\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/innovation\\\/research-academia\\\/research\\\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/innovation\\\/research-academia\\\/research\\\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/\",\"name\":\"Sony Interactive Entertainment\",\"description\":\"Pushing the Boundaries of Play\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/sonyinteractive.com\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality - Sony Interactive Entertainment","description":"New York University and Sony Interactive Entertainment research on perception-guided 3D Gaussian Splatting framework for mixed reality","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/","og_locale":"en_US","og_type":"article","og_title":"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality","og_description":"New York University and Sony Interactive Entertainment research on perception-guided 3D Gaussian Splatting framework for mixed reality","og_url":"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/","og_site_name":"Sony Interactive Entertainment","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes","Written by":"Yunxiang Zhang"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/","url":"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/","name":"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality - Sony Interactive Entertainment","isPartOf":{"@id":"https:\/\/sonyinteractive.com\/en\/#website"},"datePublished":"2026-04-22T17:47:49+00:00","description":"New York University and Sony Interactive Entertainment research on perception-guided 3D Gaussian Splatting framework for mixed reality","breadcrumb":{"@id":"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/sonyinteractive.com\/en\/innovation\/research-academia\/research\/perceptually-guided-3dgs-streaming-and-rendering-for-mixed-reality\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/sonyinteractive.com\/en\/"},{"@type":"ListItem","position":2,"name":"Perceptually Guided 3DGS Streaming and Rendering for Mixed Reality"}]},{"@type":"WebSite","@id":"https:\/\/sonyinteractive.com\/en\/#website","url":"https:\/\/sonyinteractive.com\/en\/","name":"Sony Interactive Entertainment","description":"Pushing the Boundaries of Play","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/sonyinteractive.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/research-post\/3011420","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/research-post"}],"about":[{"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/types\/research-post"}],"author":[{"embeddable":true,"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/users\/35"}],"version-history":[{"count":3,"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/research-post\/3011420\/revisions"}],"predecessor-version":[{"id":3011475,"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/research-post\/3011420\/revisions\/3011475"}],"wp:attachment":[{"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/media?parent=3011420"}],"wp:term":[{"taxonomy":"byline","embeddable":true,"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/byline?post=3011420"},{"taxonomy":"research-post-category","embeddable":true,"href":"https:\/\/sonyinteractive.com\/en\/wp-json\/wp\/v2\/research-post-category?post=3011420"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}