Importance: Patients increasingly use artificial intelligence (AI) for health-related conversations, but evaluation measures largely reflect clinician judgment. Objective: To develop and preliminarily evaluate a patient instrument assessing the quality of patient-AI health conversations. Design: Cross-sectional survey conducted June 5-14, 2026, following patient-partnered instrument development and cognitive interviews. Setting: An anonymous online survey. Participants: A convenience sample of adults with prior use of large language models for personal health-related queries, recruited from an academic medical center-affiliated Insight Panel and patient peers. Exposures: Participants' own prior health-related conversations with AI tools of their choosing. Main Outcomes and Measures: The Patient-AI Rating instrument comprised 12 communication-quality and patient engagement items rated from 1 to 5 and a separately scored patient-perceived-harm item. Internal consistency was assessed using Cronbach alpha, with sensitivity analyses using first ratings only for those using the tool to rate multiple conversations and participant-mean profiles. We examined item-total correlations, inter-item structure, and between- and within-participant variation. Results: Fourteen participants contributed 46 complete ratings; among 13 providing demographics, median age was 63 years (IQR, 53-70 years), and 8 (62%) reported female gender. Participants provided a median of 2 ratings (range, 1-17). Cronbach alpha was 0.88 (95% CI, 0.83-0.93) across all ratings, 0.85 using first ratings only, and 0.92 using participant-mean profiles. Corrected item-total correlations ranged from 0.42 to 0.72; mean inter-item correlation was 0.40. No item deletion improved Cronbach alpha. The mean quality score was 4.43 (SD, 0.45). Among 9 participants contributing multiple ratings, the intraclass correlation coefficient was 0.45, suggesting inter-conversation variability. No perceived harm was reported in 44 of 46 ratings (96%); the harm item correlated minimally with the instrument total (r = 0.09). Conclusions and Relevance: In this small cross-sectional study of engaged patients, the Patient-AI Rating instrument showed promising internal consistency and variation across rated conversations. These findings support further evaluation of patient-reported communication quality alongside clinician-based measures. Validation in larger, more diverse populations, including assessment of construct and criterion validity, is needed before broader use.